2023-03-22 17:53:31,757 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61 2023-03-22 17:53:31,770 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-22 17:53:31,785 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-22 17:53:31,786 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/cluster_5f8c7e52-8c9d-3fc1-676b-63f4f87dd676, deleteOnExit=true 2023-03-22 17:53:31,786 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-22 17:53:31,787 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/test.cache.data in system properties and HBase conf 2023-03-22 17:53:31,787 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/hadoop.tmp.dir in system properties and HBase conf 2023-03-22 17:53:31,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/hadoop.log.dir in system properties and HBase conf 2023-03-22 17:53:31,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-22 17:53:31,789 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-22 17:53:31,789 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-22 17:53:31,906 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-22 17:53:32,331 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-22 17:53:32,337 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-22 17:53:32,337 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-22 17:53:32,338 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-22 17:53:32,338 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-22 17:53:32,339 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-22 17:53:32,339 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-22 17:53:32,340 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-22 17:53:32,340 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-22 17:53:32,341 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-22 17:53:32,341 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/nfs.dump.dir in system properties and HBase conf 2023-03-22 17:53:32,341 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/java.io.tmpdir in system properties and HBase conf 2023-03-22 17:53:32,342 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-22 17:53:32,342 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-22 17:53:32,343 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-22 17:53:32,967 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-22 17:53:32,971 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-22 17:53:33,265 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-22 17:53:33,447 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-22 17:53:33,469 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-22 17:53:33,510 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-22 17:53:33,541 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/java.io.tmpdir/Jetty_localhost_36519_hdfs____.3hz6gs/webapp 2023-03-22 17:53:33,681 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36519 2023-03-22 17:53:33,697 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-22 17:53:33,698 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-22 17:53:34,260 WARN [Listener at localhost/40993] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-22 17:53:34,349 WARN [Listener at localhost/40993] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-22 17:53:34,375 WARN [Listener at localhost/40993] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-22 17:53:34,381 INFO [Listener at localhost/40993] log.Slf4jLog(67): jetty-6.1.26 2023-03-22 17:53:34,387 INFO [Listener at localhost/40993] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/java.io.tmpdir/Jetty_localhost_40211_datanode____.l8nhhf/webapp 2023-03-22 17:53:34,490 INFO [Listener at localhost/40993] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40211 2023-03-22 17:53:34,894 WARN [Listener at localhost/44491] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-22 17:53:35,437 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd1675055b3857b6f: Processing first storage report for DS-1fc815c6-96ae-423a-ae6f-c2f2833bd44d from datanode b573734f-a9bd-42cc-b584-b3b5f76a1564 2023-03-22 17:53:35,439 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd1675055b3857b6f: from storage DS-1fc815c6-96ae-423a-ae6f-c2f2833bd44d node DatanodeRegistration(127.0.0.1:37847, datanodeUuid=b573734f-a9bd-42cc-b584-b3b5f76a1564, infoPort=37457, infoSecurePort=0, ipcPort=44491, storageInfo=lv=-57;cid=testClusterID;nsid=860499070;c=1679507613043), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-22 17:53:35,439 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd1675055b3857b6f: Processing first storage report for DS-587497c4-25ee-4792-ad6f-2d991136dca7 from datanode b573734f-a9bd-42cc-b584-b3b5f76a1564 2023-03-22 17:53:35,440 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd1675055b3857b6f: from storage DS-587497c4-25ee-4792-ad6f-2d991136dca7 node DatanodeRegistration(127.0.0.1:37847, datanodeUuid=b573734f-a9bd-42cc-b584-b3b5f76a1564, infoPort=37457, infoSecurePort=0, ipcPort=44491, storageInfo=lv=-57;cid=testClusterID;nsid=860499070;c=1679507613043), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-22 17:53:35,494 DEBUG [Listener at localhost/44491] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61 2023-03-22 17:53:35,562 INFO [Listener at localhost/44491] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/cluster_5f8c7e52-8c9d-3fc1-676b-63f4f87dd676/zookeeper_0, clientPort=55285, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/cluster_5f8c7e52-8c9d-3fc1-676b-63f4f87dd676/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/cluster_5f8c7e52-8c9d-3fc1-676b-63f4f87dd676/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-22 17:53:35,577 INFO [Listener at localhost/44491] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=55285 2023-03-22 17:53:35,586 INFO [Listener at localhost/44491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-22 17:53:35,588 INFO [Listener at localhost/44491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-22 17:53:36,264 INFO [Listener at localhost/44491] util.FSUtils(471): Created version file at hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7 with version=8 2023-03-22 17:53:36,264 INFO [Listener at localhost/44491] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/hbase-staging 2023-03-22 17:53:36,596 INFO [Listener at localhost/44491] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-22 17:53:37,163 INFO [Listener at localhost/44491] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-22 17:53:37,194 INFO [Listener at localhost/44491] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-22 17:53:37,195 INFO [Listener at localhost/44491] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-22 17:53:37,195 INFO [Listener at localhost/44491] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-22 17:53:37,195 INFO [Listener at localhost/44491] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-22 17:53:37,196 INFO [Listener at localhost/44491] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-22 17:53:37,340 INFO [Listener at localhost/44491] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-22 17:53:37,422 DEBUG [Listener at localhost/44491] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-22 17:53:37,518 INFO [Listener at localhost/44491] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40203 2023-03-22 17:53:37,528 INFO [Listener at localhost/44491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-22 17:53:37,530 INFO [Listener at localhost/44491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-22 17:53:37,553 INFO [Listener at localhost/44491] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40203 connecting to ZooKeeper ensemble=127.0.0.1:55285 2023-03-22 17:53:37,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:402030x0, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-22 17:53:37,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40203-0x1010d5c59010000 connected 2023-03-22 17:53:37,625 DEBUG [Listener at localhost/44491] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-22 17:53:37,626 DEBUG [Listener at localhost/44491] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-22 17:53:37,629 DEBUG [Listener at localhost/44491] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-22 17:53:37,637 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40203 2023-03-22 17:53:37,637 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40203 2023-03-22 17:53:37,638 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40203 2023-03-22 17:53:37,638 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40203 2023-03-22 17:53:37,638 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40203 2023-03-22 17:53:37,646 INFO [Listener at localhost/44491] master.HMaster(444): hbase.rootdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7, hbase.cluster.distributed=false 2023-03-22 17:53:37,724 INFO [Listener at localhost/44491] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-22 17:53:37,724 INFO [Listener at localhost/44491] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-22 17:53:37,725 INFO [Listener at localhost/44491] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-22 17:53:37,725 INFO [Listener at localhost/44491] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-22 17:53:37,725 INFO [Listener at localhost/44491] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-22 17:53:37,725 INFO [Listener at localhost/44491] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-22 17:53:37,730 INFO [Listener at localhost/44491] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-22 17:53:37,734 INFO [Listener at localhost/44491] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34403 2023-03-22 17:53:37,736 INFO [Listener at localhost/44491] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-22 17:53:37,742 DEBUG [Listener at localhost/44491] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-22 17:53:37,743 INFO [Listener at localhost/44491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-22 17:53:37,746 INFO [Listener at localhost/44491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-22 17:53:37,749 INFO [Listener at localhost/44491] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34403 connecting to ZooKeeper ensemble=127.0.0.1:55285 2023-03-22 17:53:37,752 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:344030x0, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-22 17:53:37,753 DEBUG [Listener at localhost/44491] zookeeper.ZKUtil(164): regionserver:344030x0, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-22 17:53:37,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34403-0x1010d5c59010001 connected 2023-03-22 17:53:37,755 DEBUG [Listener at localhost/44491] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-22 17:53:37,756 DEBUG [Listener at localhost/44491] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-22 17:53:37,757 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34403 2023-03-22 17:53:37,757 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34403 2023-03-22 17:53:37,757 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34403 2023-03-22 17:53:37,759 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34403 2023-03-22 17:53:37,759 DEBUG [Listener at localhost/44491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34403 2023-03-22 17:53:37,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:53:37,775 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-22 17:53:37,776 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:53:37,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-22 17:53:37,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-22 17:53:37,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:53:37,799 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-22 17:53:37,800 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40203,1679507616414 from backup master directory 2023-03-22 17:53:37,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-22 17:53:37,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:53:37,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-22 17:53:37,804 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-22 17:53:37,804 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:53:37,808 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-22 17:53:37,809 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-22 17:53:37,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/hbase.id with ID: b51c337f-b71c-43a5-8e9e-9792274a9f3d 2023-03-22 17:53:37,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-22 17:53:37,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:53:37,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2c57173e to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:38,019 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b7c9720, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:38,042 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:53:38,044 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-22 17:53:38,060 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-22 17:53:38,060 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-22 17:53:38,062 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:53:38,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:53:38,067 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-22 17:53:38,098 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store-tmp 2023-03-22 17:53:38,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:53:38,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-22 17:53:38,525 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-22 17:53:38,526 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-22 17:53:38,526 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-22 17:53:38,526 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-22 17:53:38,526 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-22 17:53:38,526 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-22 17:53:38,528 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/WALs/jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:53:38,549 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40203%2C1679507616414, suffix=, logDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/WALs/jenkins-hbase4.apache.org,40203,1679507616414, archiveDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/oldWALs, maxLogs=10 2023-03-22 17:53:38,604 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37847,DS-1fc815c6-96ae-423a-ae6f-c2f2833bd44d,DISK] 2023-03-22 17:53:38,616 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:53:38,690 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/WALs/jenkins-hbase4.apache.org,40203,1679507616414/jenkins-hbase4.apache.org%2C40203%2C1679507616414.1679507618560 2023-03-22 17:53:38,691 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37847,DS-1fc815c6-96ae-423a-ae6f-c2f2833bd44d,DISK]] 2023-03-22 17:53:38,691 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:53:38,692 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:53:38,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-22 17:53:38,698 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-22 17:53:38,770 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-22 17:53:38,777 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-22 17:53:38,800 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-22 17:53:38,814 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:38,820 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-22 17:53:38,822 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-22 17:53:38,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-22 17:53:38,844 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:53:38,846 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62123574, jitterRate=-0.07428660988807678}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-22 17:53:38,846 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-22 17:53:38,848 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-22 17:53:38,868 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-22 17:53:38,868 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-22 17:53:38,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-22 17:53:38,872 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-22 17:53:38,905 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-03-22 17:53:38,905 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-22 17:53:38,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-22 17:53:38,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-22 17:53:38,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-22 17:53:38,969 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-22 17:53:38,971 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-22 17:53:38,976 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-22 17:53:38,980 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-22 17:53:38,983 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:53:38,985 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-22 17:53:38,985 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-22 17:53:38,997 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-22 17:53:39,003 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-22 17:53:39,003 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-22 17:53:39,003 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:53:39,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,40203,1679507616414, sessionid=0x1010d5c59010000, setting cluster-up flag (Was=false) 2023-03-22 17:53:39,019 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:53:39,025 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-22 17:53:39,026 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:53:39,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:53:39,039 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-22 17:53:39,041 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:53:39,043 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.hbase-snapshot/.tmp 2023-03-22 17:53:39,063 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(952): ClusterId : b51c337f-b71c-43a5-8e9e-9792274a9f3d 2023-03-22 17:53:39,069 DEBUG [RS:0;jenkins-hbase4:34403] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-22 17:53:39,077 DEBUG [RS:0;jenkins-hbase4:34403] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-22 17:53:39,077 DEBUG [RS:0;jenkins-hbase4:34403] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-22 17:53:39,084 DEBUG [RS:0;jenkins-hbase4:34403] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-22 17:53:39,085 DEBUG [RS:0;jenkins-hbase4:34403] zookeeper.ReadOnlyZKClient(139): Connect 0x17c04ca6 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:39,099 DEBUG [RS:0;jenkins-hbase4:34403] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1907a02e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:39,100 DEBUG [RS:0;jenkins-hbase4:34403] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d3827d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-22 17:53:39,128 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34403 2023-03-22 17:53:39,133 INFO [RS:0;jenkins-hbase4:34403] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-22 17:53:39,134 INFO [RS:0;jenkins-hbase4:34403] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-22 17:53:39,134 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1023): About to register with Master. 2023-03-22 17:53:39,137 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40203,1679507616414 with isa=jenkins-hbase4.apache.org/172.31.14.131:34403, startcode=1679507617723 2023-03-22 17:53:39,157 DEBUG [RS:0;jenkins-hbase4:34403] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-22 17:53:39,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-22 17:53:39,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-22 17:53:39,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-22 17:53:39,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-22 17:53:39,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-22 17:53:39,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-22 17:53:39,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-22 17:53:39,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,229 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679507649229 2023-03-22 17:53:39,231 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58401, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-22 17:53:39,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-22 17:53:39,233 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-22 17:53:39,233 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-22 17:53:39,242 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-22 17:53:39,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:39,247 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-22 17:53:39,254 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-22 17:53:39,255 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-22 17:53:39,255 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-22 17:53:39,255 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-22 17:53:39,256 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,257 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-22 17:53:39,258 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-22 17:53:39,259 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-22 17:53:39,261 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-22 17:53:39,262 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-22 17:53:39,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679507619263,5,FailOnTimeoutGroup] 2023-03-22 17:53:39,264 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679507619263,5,FailOnTimeoutGroup] 2023-03-22 17:53:39,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-22 17:53:39,265 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,266 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,274 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(2829): Master is not running yet 2023-03-22 17:53:39,275 WARN [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-03-22 17:53:39,376 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40203,1679507616414 with isa=jenkins-hbase4.apache.org/172.31.14.131:34403, startcode=1679507617723 2023-03-22 17:53:39,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:39,390 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7 2023-03-22 17:53:39,391 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40993 2023-03-22 17:53:39,391 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-22 17:53:39,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-22 17:53:39,396 DEBUG [RS:0;jenkins-hbase4:34403] zookeeper.ZKUtil(162): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:39,396 WARN [RS:0;jenkins-hbase4:34403] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-22 17:53:39,396 INFO [RS:0;jenkins-hbase4:34403] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-22 17:53:39,397 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/WALs/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:39,398 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34403,1679507617723] 2023-03-22 17:53:39,405 DEBUG [RS:0;jenkins-hbase4:34403] zookeeper.ZKUtil(162): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:39,415 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-22 17:53:39,422 INFO [RS:0;jenkins-hbase4:34403] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-22 17:53:39,440 INFO [RS:0;jenkins-hbase4:34403] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-22 17:53:39,444 INFO [RS:0;jenkins-hbase4:34403] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-22 17:53:39,444 INFO [RS:0;jenkins-hbase4:34403] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,445 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-22 17:53:39,451 INFO [RS:0;jenkins-hbase4:34403] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,451 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,452 DEBUG [RS:0;jenkins-hbase4:34403] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-22 17:53:39,453 INFO [RS:0;jenkins-hbase4:34403] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,453 INFO [RS:0;jenkins-hbase4:34403] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,454 INFO [RS:0;jenkins-hbase4:34403] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,469 INFO [RS:0;jenkins-hbase4:34403] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-22 17:53:39,472 INFO [RS:0;jenkins-hbase4:34403] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34403,1679507617723-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:39,487 INFO [RS:0;jenkins-hbase4:34403] regionserver.Replication(203): jenkins-hbase4.apache.org,34403,1679507617723 started 2023-03-22 17:53:39,487 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34403,1679507617723, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34403, sessionid=0x1010d5c59010001 2023-03-22 17:53:39,487 DEBUG [RS:0;jenkins-hbase4:34403] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-22 17:53:39,487 DEBUG [RS:0;jenkins-hbase4:34403] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:39,487 DEBUG [RS:0;jenkins-hbase4:34403] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34403,1679507617723' 2023-03-22 17:53:39,487 DEBUG [RS:0;jenkins-hbase4:34403] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:39,488 DEBUG [RS:0;jenkins-hbase4:34403] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:39,488 DEBUG [RS:0;jenkins-hbase4:34403] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-22 17:53:39,488 DEBUG [RS:0;jenkins-hbase4:34403] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-22 17:53:39,488 DEBUG [RS:0;jenkins-hbase4:34403] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:39,488 DEBUG [RS:0;jenkins-hbase4:34403] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34403,1679507617723' 2023-03-22 17:53:39,489 DEBUG [RS:0;jenkins-hbase4:34403] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-22 17:53:39,489 DEBUG [RS:0;jenkins-hbase4:34403] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-22 17:53:39,489 DEBUG [RS:0;jenkins-hbase4:34403] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-22 17:53:39,489 INFO [RS:0;jenkins-hbase4:34403] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-22 17:53:39,490 INFO [RS:0;jenkins-hbase4:34403] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-22 17:53:39,600 INFO [RS:0;jenkins-hbase4:34403] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34403%2C1679507617723, suffix=, logDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/WALs/jenkins-hbase4.apache.org,34403,1679507617723, archiveDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/oldWALs, maxLogs=32 2023-03-22 17:53:39,618 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37847,DS-1fc815c6-96ae-423a-ae6f-c2f2833bd44d,DISK] 2023-03-22 17:53:39,622 INFO [RS:0;jenkins-hbase4:34403] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/WALs/jenkins-hbase4.apache.org,34403,1679507617723/jenkins-hbase4.apache.org%2C34403%2C1679507617723.1679507619603 2023-03-22 17:53:39,622 DEBUG [RS:0;jenkins-hbase4:34403] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37847,DS-1fc815c6-96ae-423a-ae6f-c2f2833bd44d,DISK]] 2023-03-22 17:53:39,682 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-22 17:53:39,683 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-22 17:53:39,683 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7 2023-03-22 17:53:39,699 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:53:39,701 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-22 17:53:39,704 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/info 2023-03-22 17:53:39,704 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-22 17:53:39,705 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:39,706 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-22 17:53:39,708 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/rep_barrier 2023-03-22 17:53:39,709 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-22 17:53:39,709 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:39,710 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-22 17:53:39,711 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/table 2023-03-22 17:53:39,712 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-22 17:53:39,713 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:39,714 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740 2023-03-22 17:53:39,715 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740 2023-03-22 17:53:39,718 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:53:39,720 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-22 17:53:39,723 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:53:39,724 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60998169, jitterRate=-0.0910564512014389}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:53:39,724 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-22 17:53:39,724 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-22 17:53:39,725 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-22 17:53:39,725 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-22 17:53:39,725 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-22 17:53:39,725 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-22 17:53:39,726 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-22 17:53:39,726 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-22 17:53:39,732 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-22 17:53:39,732 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-22 17:53:39,740 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-22 17:53:39,752 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-22 17:53:39,754 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-22 17:53:39,906 DEBUG [jenkins-hbase4:40203] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-22 17:53:39,909 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34403,1679507617723, state=OPENING 2023-03-22 17:53:39,916 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-22 17:53:39,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:53:39,918 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-22 17:53:39,920 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:53:40,105 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:40,107 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-22 17:53:40,111 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42716, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-22 17:53:40,123 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-22 17:53:40,124 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-22 17:53:40,127 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34403%2C1679507617723.meta, suffix=.meta, logDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/WALs/jenkins-hbase4.apache.org,34403,1679507617723, archiveDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/oldWALs, maxLogs=32 2023-03-22 17:53:40,146 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37847,DS-1fc815c6-96ae-423a-ae6f-c2f2833bd44d,DISK] 2023-03-22 17:53:40,151 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/WALs/jenkins-hbase4.apache.org,34403,1679507617723/jenkins-hbase4.apache.org%2C34403%2C1679507617723.meta.1679507620129.meta 2023-03-22 17:53:40,152 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37847,DS-1fc815c6-96ae-423a-ae6f-c2f2833bd44d,DISK]] 2023-03-22 17:53:40,152 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:53:40,154 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-22 17:53:40,175 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-22 17:53:40,181 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-22 17:53:40,188 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-22 17:53:40,188 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:53:40,189 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-22 17:53:40,189 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-22 17:53:40,192 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-22 17:53:40,194 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/info 2023-03-22 17:53:40,194 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/info 2023-03-22 17:53:40,195 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-22 17:53:40,196 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:40,196 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-22 17:53:40,197 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/rep_barrier 2023-03-22 17:53:40,198 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/rep_barrier 2023-03-22 17:53:40,198 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-22 17:53:40,199 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:40,199 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-22 17:53:40,201 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/table 2023-03-22 17:53:40,201 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/table 2023-03-22 17:53:40,201 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-22 17:53:40,202 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:40,204 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740 2023-03-22 17:53:40,206 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740 2023-03-22 17:53:40,209 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:53:40,212 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-22 17:53:40,213 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69293404, jitterRate=0.03255218267440796}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:53:40,213 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-22 17:53:40,222 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679507620098 2023-03-22 17:53:40,240 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-22 17:53:40,241 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-22 17:53:40,242 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34403,1679507617723, state=OPEN 2023-03-22 17:53:40,245 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-22 17:53:40,245 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-22 17:53:40,251 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-22 17:53:40,251 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34403,1679507617723 in 325 msec 2023-03-22 17:53:40,256 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-22 17:53:40,256 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 513 msec 2023-03-22 17:53:40,262 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1500 sec 2023-03-22 17:53:40,262 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679507620262, completionTime=-1 2023-03-22 17:53:40,263 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-22 17:53:40,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-22 17:53:40,332 DEBUG [hconnection-0x2a4b69b2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:40,336 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42720, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:40,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-22 17:53:40,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679507680356 2023-03-22 17:53:40,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679507740357 2023-03-22 17:53:40,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 93 msec 2023-03-22 17:53:40,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40203,1679507616414-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:40,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40203,1679507616414-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:40,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40203,1679507616414-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:40,385 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40203, period=300000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:40,386 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-22 17:53:40,393 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-22 17:53:40,402 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-22 17:53:40,403 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-22 17:53:40,412 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-22 17:53:40,415 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-22 17:53:40,417 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-22 17:53:40,438 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:40,440 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096 empty. 2023-03-22 17:53:40,441 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:40,441 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-22 17:53:40,862 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-22 17:53:40,865 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => e8ac5092d76a6dd1e148cbdfe0f07096, NAME => 'hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp 2023-03-22 17:53:40,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:53:40,880 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing e8ac5092d76a6dd1e148cbdfe0f07096, disabling compactions & flushes 2023-03-22 17:53:40,880 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:53:40,880 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:53:40,880 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. after waiting 0 ms 2023-03-22 17:53:40,880 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:53:40,880 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:53:40,880 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for e8ac5092d76a6dd1e148cbdfe0f07096: 2023-03-22 17:53:40,885 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-22 17:53:40,900 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679507620888"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507620888"}]},"ts":"1679507620888"} 2023-03-22 17:53:40,926 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-22 17:53:40,928 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-22 17:53:40,932 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507620928"}]},"ts":"1679507620928"} 2023-03-22 17:53:40,935 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-22 17:53:40,945 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e8ac5092d76a6dd1e148cbdfe0f07096, ASSIGN}] 2023-03-22 17:53:40,948 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e8ac5092d76a6dd1e148cbdfe0f07096, ASSIGN 2023-03-22 17:53:40,949 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e8ac5092d76a6dd1e148cbdfe0f07096, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=false 2023-03-22 17:53:41,100 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e8ac5092d76a6dd1e148cbdfe0f07096, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:41,101 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679507621100"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507621100"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507621100"}]},"ts":"1679507621100"} 2023-03-22 17:53:41,105 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e8ac5092d76a6dd1e148cbdfe0f07096, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:53:41,265 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:53:41,266 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e8ac5092d76a6dd1e148cbdfe0f07096, NAME => 'hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:53:41,267 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:41,267 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:53:41,267 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:41,267 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:41,269 INFO [StoreOpener-e8ac5092d76a6dd1e148cbdfe0f07096-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:41,271 DEBUG [StoreOpener-e8ac5092d76a6dd1e148cbdfe0f07096-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096/info 2023-03-22 17:53:41,271 DEBUG [StoreOpener-e8ac5092d76a6dd1e148cbdfe0f07096-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096/info 2023-03-22 17:53:41,272 INFO [StoreOpener-e8ac5092d76a6dd1e148cbdfe0f07096-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8ac5092d76a6dd1e148cbdfe0f07096 columnFamilyName info 2023-03-22 17:53:41,273 INFO [StoreOpener-e8ac5092d76a6dd1e148cbdfe0f07096-1] regionserver.HStore(310): Store=e8ac5092d76a6dd1e148cbdfe0f07096/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:41,274 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:41,275 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:41,279 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:53:41,283 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:53:41,284 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e8ac5092d76a6dd1e148cbdfe0f07096; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71763112, jitterRate=0.06935369968414307}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-22 17:53:41,284 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e8ac5092d76a6dd1e148cbdfe0f07096: 2023-03-22 17:53:41,286 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096., pid=6, masterSystemTime=1679507621259 2023-03-22 17:53:41,290 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:53:41,290 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:53:41,291 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e8ac5092d76a6dd1e148cbdfe0f07096, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:41,291 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679507621290"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507621290"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507621290"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507621290"}]},"ts":"1679507621290"} 2023-03-22 17:53:41,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-22 17:53:41,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e8ac5092d76a6dd1e148cbdfe0f07096, server=jenkins-hbase4.apache.org,34403,1679507617723 in 190 msec 2023-03-22 17:53:41,301 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-22 17:53:41,302 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e8ac5092d76a6dd1e148cbdfe0f07096, ASSIGN in 353 msec 2023-03-22 17:53:41,303 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-22 17:53:41,303 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507621303"}]},"ts":"1679507621303"} 2023-03-22 17:53:41,306 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-22 17:53:41,310 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-22 17:53:41,313 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 906 msec 2023-03-22 17:53:41,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-22 17:53:41,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-22 17:53:41,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:53:41,350 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-22 17:53:41,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-22 17:53:41,372 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 28 msec 2023-03-22 17:53:41,394 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-22 17:53:41,418 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-22 17:53:41,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 29 msec 2023-03-22 17:53:41,442 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-22 17:53:41,445 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-22 17:53:41,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.640sec 2023-03-22 17:53:41,447 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-22 17:53:41,449 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-22 17:53:41,449 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-22 17:53:41,450 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40203,1679507616414-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-22 17:53:41,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40203,1679507616414-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-22 17:53:41,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-22 17:53:41,469 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:41,474 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:41,491 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:41,507 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42726, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:41,519 INFO [Listener at localhost/44491] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:53:41,540 INFO [Listener at localhost/44491] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=282, ProcessCount=169, AvailableMemoryMB=6720 2023-03-22 17:53:41,558 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-22 17:53:41,562 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46540, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-22 17:53:41,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:53:41,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:53:41,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-22 17:53:41,579 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-22 17:53:41,581 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-22 17:53:41,584 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:41,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-22 17:53:41,586 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6 empty. 2023-03-22 17:53:41,587 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:41,587 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:53:41,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-22 17:53:41,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-22 17:53:41,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-22 17:53:42,005 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:53:42,007 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 00ab0c7e213b0d56501a0c20a5e0b9c6, NAME => 'TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp 2023-03-22 17:53:42,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-22 17:53:42,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:53:42,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 00ab0c7e213b0d56501a0c20a5e0b9c6, disabling compactions & flushes 2023-03-22 17:53:42,420 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:42,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:42,420 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. after waiting 0 ms 2023-03-22 17:53:42,421 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:42,421 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:42,421 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:42,425 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-22 17:53:42,427 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507622426"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507622426"}]},"ts":"1679507622426"} 2023-03-22 17:53:42,429 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-22 17:53:42,431 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-22 17:53:42,431 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507622431"}]},"ts":"1679507622431"} 2023-03-22 17:53:42,434 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-22 17:53:42,438 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ab0c7e213b0d56501a0c20a5e0b9c6, ASSIGN}] 2023-03-22 17:53:42,440 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ab0c7e213b0d56501a0c20a5e0b9c6, ASSIGN 2023-03-22 17:53:42,441 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ab0c7e213b0d56501a0c20a5e0b9c6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=false 2023-03-22 17:53:42,592 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=00ab0c7e213b0d56501a0c20a5e0b9c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:42,593 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507622592"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507622592"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507622592"}]},"ts":"1679507622592"} 2023-03-22 17:53:42,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:53:42,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-22 17:53:42,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:42,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 00ab0c7e213b0d56501a0c20a5e0b9c6, NAME => 'TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:53:42,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:53:42,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,758 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,760 DEBUG [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A 2023-03-22 17:53:42,760 DEBUG [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A 2023-03-22 17:53:42,761 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:53:42,762 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00ab0c7e213b0d56501a0c20a5e0b9c6 columnFamilyName A 2023-03-22 17:53:42,762 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.HStore(310): Store=00ab0c7e213b0d56501a0c20a5e0b9c6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:42,763 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,764 DEBUG [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B 2023-03-22 17:53:42,764 DEBUG [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B 2023-03-22 17:53:42,765 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:53:42,765 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00ab0c7e213b0d56501a0c20a5e0b9c6 columnFamilyName B 2023-03-22 17:53:42,765 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.HStore(310): Store=00ab0c7e213b0d56501a0c20a5e0b9c6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:42,766 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,767 DEBUG [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C 2023-03-22 17:53:42,768 DEBUG [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C 2023-03-22 17:53:42,768 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:53:42,768 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00ab0c7e213b0d56501a0c20a5e0b9c6 columnFamilyName C 2023-03-22 17:53:42,769 INFO [StoreOpener-00ab0c7e213b0d56501a0c20a5e0b9c6-1] regionserver.HStore(310): Store=00ab0c7e213b0d56501a0c20a5e0b9c6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:53:42,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:42,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:53:42,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:42,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:53:42,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 00ab0c7e213b0d56501a0c20a5e0b9c6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71453188, jitterRate=0.06473547220230103}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:53:42,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:42,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., pid=11, masterSystemTime=1679507622749 2023-03-22 17:53:42,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:42,785 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:42,785 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=00ab0c7e213b0d56501a0c20a5e0b9c6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:42,786 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507622785"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507622785"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507622785"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507622785"}]},"ts":"1679507622785"} 2023-03-22 17:53:42,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-22 17:53:42,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 in 193 msec 2023-03-22 17:53:42,795 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-22 17:53:42,795 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ab0c7e213b0d56501a0c20a5e0b9c6, ASSIGN in 354 msec 2023-03-22 17:53:42,796 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-22 17:53:42,797 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507622796"}]},"ts":"1679507622796"} 2023-03-22 17:53:42,799 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-22 17:53:42,839 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-22 17:53:42,842 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2640 sec 2023-03-22 17:53:43,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-22 17:53:43,710 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-22 17:53:43,715 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x44ef43e3 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,720 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2678d495, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,722 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,725 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45278, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,728 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-22 17:53:43,730 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48672, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-22 17:53:43,738 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x61e72c30 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,742 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b886c8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,743 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,748 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,749 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,755 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,755 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,763 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,763 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x63859c62 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,770 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,771 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,776 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,776 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,796 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,797 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,805 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,805 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:53:43,809 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:53:43,822 DEBUG [hconnection-0x1e045d2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,824 DEBUG [hconnection-0x7d721dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,827 DEBUG [hconnection-0x156ee019-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,827 DEBUG [hconnection-0x7576e4ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:43,836 DEBUG [hconnection-0x3e1787eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,841 DEBUG [hconnection-0x6d34219d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,841 DEBUG [hconnection-0x7c8e030b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,841 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45294, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,843 DEBUG [hconnection-0x62a93ad6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,842 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,846 DEBUG [hconnection-0x63093886-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:53:43,852 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45302, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,854 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45324, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,855 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45314, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,859 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45338, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:43,868 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45342, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,880 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:43,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:43,882 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45348, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,888 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45350, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:53:43,891 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:43,891 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:43,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:43,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:43,901 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:43,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:43,901 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:43,902 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,902 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:43,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:43,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:43,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:43,930 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:43,939 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:43,939 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:43,945 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:43,945 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:43,956 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:53:43,956 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:43,957 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:43,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:43,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:43,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,961 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:43,961 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:43,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:43,964 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:43,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:43,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:43,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:43,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:43,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:43,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:43,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:43,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:43,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:43,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:43,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae6119e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:43,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:43,981 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:43,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:43,981 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:43,986 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:43,986 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,986 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:43,987 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:43,992 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:43,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:43,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:43,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:43,992 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:43,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:44,006 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,006 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:44,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,006 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:44,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,011 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:44,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:44,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:44,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:44,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:44,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:44,028 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:44,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,030 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,030 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:44,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:44,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:44,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507684085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507624065, max timestamp=1679507624089Num uniques 859; 2023-03-22 17:53:44,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:44,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507624065, max timestamp=1679507624108Num uniques 859; 2023-03-22 17:53:44,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:44,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507624065, max timestamp=1679507624105Num uniques 859; 2023-03-22 17:53:44,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:44,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:44,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:44,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:44,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:44,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:44,205 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:44,205 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:44,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:44,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:44,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:44,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:44,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:44,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507684195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507684198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507684203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:44,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:44,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:44,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507684207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:44,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:44,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:44,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:44,221 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:44,221 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:44,221 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:44,222 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:44,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:44,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:44,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,223 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:44,223 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:44,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507684224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,228 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,230 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:44,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6d1887[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:44,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:44,234 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,235 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,236 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,236 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:44,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:44,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:44,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:44,239 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:44,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:44,251 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:44,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:44,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:44,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:44,251 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:44,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,264 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,274 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,274 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,274 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:44,274 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,275 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:44,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:44,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:44,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:44,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:44,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:44,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:44,277 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:44,277 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:44,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507684316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507684318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507684327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507684317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,377 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:44,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:44,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:44,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:44,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:44,398 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:44,398 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:44,399 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:44,399 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:44,400 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,400 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:44,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:44,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:44,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:44,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:44,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:44,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:44,437 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:44,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:44,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:44,440 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:44,441 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:44,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:44,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:44,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507684447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,455 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:44,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fa309a8[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:44,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:44,460 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:44,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:44,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:44,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:44,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:44,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:44,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:44,469 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:44,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:44,476 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:44,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:44,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:44,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:44,476 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:44,478 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,478 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:44,479 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:44,480 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:44,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,494 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:44,494 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,494 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,494 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:44,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:44,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:44,494 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:44,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:44,495 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:44,495 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:44,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:44,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507684524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507684529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507684532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507684537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,567 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/7f8d3f8eadc240b8b731b84694bd83c6 2023-03-22 17:53:44,596 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:44,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:44,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:44,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:44,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:44,612 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:44,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:44,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:44,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:44,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:44,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:44,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:44,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:44,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:44,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:44,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:44,622 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:44,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:44,626 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:44,626 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:44,626 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:44,626 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:44,626 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:44,626 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:44,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:44,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d1218e6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:44,643 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:44,643 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,645 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:44,646 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,646 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:44,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:44,657 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:44,657 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:44,657 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:44,657 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:44,657 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:44,666 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,666 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,666 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:44,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:44,674 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,677 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:44,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/4ce32607e9cd400aa3dc7f1b3829aa74 2023-03-22 17:53:44,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:44,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:44,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:44,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,686 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:44,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:44,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:44,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:44,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:44,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,687 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:44,687 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:44,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:44,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/b566ab03a3f742c893524ae180b46a3d 2023-03-22 17:53:44,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/7f8d3f8eadc240b8b731b84694bd83c6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7f8d3f8eadc240b8b731b84694bd83c6 2023-03-22 17:53:44,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7f8d3f8eadc240b8b731b84694bd83c6, entries=150, sequenceid=11, filesize=11.6 K 2023-03-22 17:53:44,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/4ce32607e9cd400aa3dc7f1b3829aa74 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4ce32607e9cd400aa3dc7f1b3829aa74 2023-03-22 17:53:44,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507684754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4ce32607e9cd400aa3dc7f1b3829aa74, entries=150, sequenceid=11, filesize=11.6 K 2023-03-22 17:53:44,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/b566ab03a3f742c893524ae180b46a3d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b566ab03a3f742c893524ae180b46a3d 2023-03-22 17:53:44,787 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:44,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:44,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:44,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b566ab03a3f742c893524ae180b46a3d, entries=150, sequenceid=11, filesize=11.6 K 2023-03-22 17:53:44,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:44,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:44,813 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:44,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:44,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:44,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:44,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,815 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 859ms, sequenceid=11, compaction requested=false 2023-03-22 17:53:44,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,817 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:44,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:44,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:44,818 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-22 17:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:44,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:44,821 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:44,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:44,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:44,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:44,825 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:44,828 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:44,828 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:44,828 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-22 17:53:44,829 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:44,829 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:44,829 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:44,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:44,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507684892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507684899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:44,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507684902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507684900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:44,937 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/1211ad3865b34110999773e9edb03ca3 2023-03-22 17:53:45,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507685014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507685019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507685022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507685024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507685239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507685238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507685247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507685248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507685259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,417 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/4a95cdda7ef346a3a9d7d8c9d611d1ae 2023-03-22 17:53:45,500 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/b0582734ba5d498fbf1d25eae2eb0c6f 2023-03-22 17:53:45,523 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/1211ad3865b34110999773e9edb03ca3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1211ad3865b34110999773e9edb03ca3 2023-03-22 17:53:45,539 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1211ad3865b34110999773e9edb03ca3, entries=150, sequenceid=48, filesize=11.6 K 2023-03-22 17:53:45,541 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/4a95cdda7ef346a3a9d7d8c9d611d1ae as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4a95cdda7ef346a3a9d7d8c9d611d1ae 2023-03-22 17:53:45,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507685552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507685553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507685557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,563 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4a95cdda7ef346a3a9d7d8c9d611d1ae, entries=150, sequenceid=48, filesize=11.6 K 2023-03-22 17:53:45,565 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/b0582734ba5d498fbf1d25eae2eb0c6f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b0582734ba5d498fbf1d25eae2eb0c6f 2023-03-22 17:53:45,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:45,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507685562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,584 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b0582734ba5d498fbf1d25eae2eb0c6f, entries=150, sequenceid=48, filesize=11.6 K 2023-03-22 17:53:45,592 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 764ms, sequenceid=48, compaction requested=false 2023-03-22 17:53:45,593 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:45,594 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:45,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:45,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:45,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:45,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:45,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:45,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:45,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:45,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:45,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5702963b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:45,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:45,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:45,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:45,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:45,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:45,605 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:45,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:45,611 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:45,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:45,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:45,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:45,611 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:45,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:45,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:45,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:45,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:45,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:45,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:45,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:45,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:45,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:45,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:45,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:45,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:45,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:45,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:45,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:45,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:45,632 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:45,632 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:45,732 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:45,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:45,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:45,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:45,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:45,747 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:45,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:45,753 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:45,753 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:45,762 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:45,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:45,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:45,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:45,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:45,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:45,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:45,770 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-22 17:53:45,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:45,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:45,775 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:45,775 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:45,779 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:45,781 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:45,781 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:45,781 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:45,781 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:45,781 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:53:45,782 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:45,782 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:45,782 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:45,782 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:45,784 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:45,784 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:45,875 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/604f49fc72794b769c2a8020fa66acfb 2023-03-22 17:53:45,920 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-22 17:53:45,929 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-22 17:53:45,934 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/bc8db7073f5848f5a094696418ee3919 2023-03-22 17:53:45,989 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/1b857a0f0153429ab94faf0e8a510924 2023-03-22 17:53:46,004 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/604f49fc72794b769c2a8020fa66acfb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/604f49fc72794b769c2a8020fa66acfb 2023-03-22 17:53:46,017 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/604f49fc72794b769c2a8020fa66acfb, entries=100, sequenceid=58, filesize=9.3 K 2023-03-22 17:53:46,020 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/bc8db7073f5848f5a094696418ee3919 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/bc8db7073f5848f5a094696418ee3919 2023-03-22 17:53:46,042 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/bc8db7073f5848f5a094696418ee3919, entries=100, sequenceid=58, filesize=9.3 K 2023-03-22 17:53:46,046 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/1b857a0f0153429ab94faf0e8a510924 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1b857a0f0153429ab94faf0e8a510924 2023-03-22 17:53:46,061 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1b857a0f0153429ab94faf0e8a510924, entries=100, sequenceid=58, filesize=9.3 K 2023-03-22 17:53:46,064 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 282ms, sequenceid=58, compaction requested=true 2023-03-22 17:53:46,064 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:46,064 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,064 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:46,065 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:46,065 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,065 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:46,065 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:46,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:46,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@502c157c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:46,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:46,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,076 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:46,076 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:46,077 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:46,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:46,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:46,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:46,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:46,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:46,091 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:46,091 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:46,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:46,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:46,092 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:46,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:46,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:53:46,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:46,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:46,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:46,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:46,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:46,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:46,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:46,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:46,096 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,101 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,107 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,107 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,107 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:46,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:46,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:46,107 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:46,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:46,107 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:46,108 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:46,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:46,110 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:46,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:46,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/a9c0239a3b6141b0bd7e6e135d55fabf 2023-03-22 17:53:46,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507686161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679507626147, max timestamp=1679507626160Num uniques 859; 2023-03-22 17:53:46,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7050, suffix heap size=117064,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=154 2023-03-22 17:53:46,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507626147, max timestamp=1679507626160Num uniques 859; 2023-03-22 17:53:46,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:46,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679507626147, max timestamp=1679507626160Num uniques 859; 2023-03-22 17:53:46,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-22 17:53:46,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/3d653c9ff3fe4455a4943fec1aa1caea 2023-03-22 17:53:46,210 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:46,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:46,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:46,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:46,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:46,220 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:46,221 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:46,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:46,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:46,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,225 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:46,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:46,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:46,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:46,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:46,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:46,238 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,239 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:46,239 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:46,239 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:46,240 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:46,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:46,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,243 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:46,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a5a898[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:46,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:46,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,251 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:46,252 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507626197, max timestamp=1679507626244Num uniques 1013; 2023-03-22 17:53:46,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46393, new segment data size=12268, suffix heap size=133320,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=268 2023-03-22 17:53:46,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507686241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,252 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:46,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507626197, max timestamp=1679507626244Num uniques 1018; 2023-03-22 17:53:46,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-22 17:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507626197, max timestamp=1679507626244Num uniques 1009; 2023-03-22 17:53:46,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-03-22 17:53:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:46,271 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:46,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:46,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:46,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:46,271 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:46,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:46,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:46,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:46,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:46,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,287 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:46,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:46,296 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:46,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:46,296 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,297 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:46,297 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507686315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507686333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507686340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507686340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507686354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,397 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:46,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:46,406 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:46,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:46,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:46,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:46,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,411 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,411 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:46,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:46,412 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,412 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:46,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:46,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:46,415 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:46,415 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:46,416 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:46,419 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,419 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:46,419 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:46,419 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:46,429 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:46,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@acd43a2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:46,433 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:46,433 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,435 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:46,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,436 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:46,437 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:46,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:46,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:46,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:46,444 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:46,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:46,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:46,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:46,444 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:46,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507686443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:46,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507686447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507686457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507686458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:46,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:46,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:46,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:46,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:46,467 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:46,468 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:46,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:46,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:46,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507686567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:46,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:46,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:46,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:46,584 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:46,584 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:46,585 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:46,585 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:46,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:46,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:46,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:46,591 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-22 17:53:46,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:46,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:46,592 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-22 17:53:46,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:46,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:46,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:46,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:46,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:46,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:46,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,600 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:46,600 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:46,600 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,601 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:46,601 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:46,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:46,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:46,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@493dbf8e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:46,609 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:46,609 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,611 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:46,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:46,617 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:46,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:46,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:46,633 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:46,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:46,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:46,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:46,633 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:46,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:46,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:46,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:46,642 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:46,645 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,651 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:46,651 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:46,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:46,651 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:46,651 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:46,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:46,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:46,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507686649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,656 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:46,656 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:46,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507686655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/dd3281f0e8f04332808f0f41f076fa4e 2023-03-22 17:53:46,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507686665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507686668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/a9c0239a3b6141b0bd7e6e135d55fabf as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a9c0239a3b6141b0bd7e6e135d55fabf 2023-03-22 17:53:46,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a9c0239a3b6141b0bd7e6e135d55fabf, entries=150, sequenceid=68, filesize=11.6 K 2023-03-22 17:53:46,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/3d653c9ff3fe4455a4943fec1aa1caea as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3d653c9ff3fe4455a4943fec1aa1caea 2023-03-22 17:53:46,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3d653c9ff3fe4455a4943fec1aa1caea, entries=150, sequenceid=68, filesize=11.6 K 2023-03-22 17:53:46,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/dd3281f0e8f04332808f0f41f076fa4e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/dd3281f0e8f04332808f0f41f076fa4e 2023-03-22 17:53:46,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/dd3281f0e8f04332808f0f41f076fa4e, entries=150, sequenceid=68, filesize=11.6 K 2023-03-22 17:53:46,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 659ms, sequenceid=68, compaction requested=true 2023-03-22 17:53:46,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:46,755 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:46,755 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:46,755 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:46,755 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:53:46,755 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:53:46,756 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:46,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:46,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:46,767 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:53:46,767 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:53:46,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:46,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:46,768 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:46,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:46,769 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:46,769 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:46,770 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:53:46,770 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,771 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,771 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,771 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:53:46,771 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4ce32607e9cd400aa3dc7f1b3829aa74, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4a95cdda7ef346a3a9d7d8c9d611d1ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/bc8db7073f5848f5a094696418ee3919, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3d653c9ff3fe4455a4943fec1aa1caea] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=44.1 K 2023-03-22 17:53:46,771 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,771 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7f8d3f8eadc240b8b731b84694bd83c6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1211ad3865b34110999773e9edb03ca3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/604f49fc72794b769c2a8020fa66acfb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a9c0239a3b6141b0bd7e6e135d55fabf] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=44.1 K 2023-03-22 17:53:46,773 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4ce32607e9cd400aa3dc7f1b3829aa74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679507623856 2023-03-22 17:53:46,776 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 7f8d3f8eadc240b8b731b84694bd83c6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679507623856 2023-03-22 17:53:46,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:46,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:46,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:46,777 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4a95cdda7ef346a3a9d7d8c9d611d1ae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679507624177 2023-03-22 17:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,778 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1211ad3865b34110999773e9edb03ca3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679507624177 2023-03-22 17:53:46,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:46,778 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting bc8db7073f5848f5a094696418ee3919, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1679507624891 2023-03-22 17:53:46,779 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 604f49fc72794b769c2a8020fa66acfb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1679507624891 2023-03-22 17:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:46,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:46,779 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3d653c9ff3fe4455a4943fec1aa1caea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679507626078 2023-03-22 17:53:46,779 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a9c0239a3b6141b0bd7e6e135d55fabf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679507626078 2023-03-22 17:53:46,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:46,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:46,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:46,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:46,783 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:46,783 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:46,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:46,788 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=134.18 KB heapSize=374.77 KB 2023-03-22 17:53:46,788 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:46,788 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:46,788 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:46,789 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:46,789 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:46,789 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:46,822 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#13 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:46,839 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#14 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:46,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:46,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:46,948 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/60549289b32647ad90e156f379b72366 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/60549289b32647ad90e156f379b72366 2023-03-22 17:53:46,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507686958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507686961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507686971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507686975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:46,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507686975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:46,986 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 60549289b32647ad90e156f379b72366(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:46,986 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:46,986 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=12, startTime=1679507626755; duration=0sec 2023-03-22 17:53:46,987 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:46,987 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:53:46,999 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:53:46,999 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:53:46,999 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:46,999 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b566ab03a3f742c893524ae180b46a3d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b0582734ba5d498fbf1d25eae2eb0c6f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1b857a0f0153429ab94faf0e8a510924, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/dd3281f0e8f04332808f0f41f076fa4e] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=44.1 K 2023-03-22 17:53:47,004 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b566ab03a3f742c893524ae180b46a3d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679507623856 2023-03-22 17:53:47,005 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b0582734ba5d498fbf1d25eae2eb0c6f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679507624177 2023-03-22 17:53:47,010 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1b857a0f0153429ab94faf0e8a510924, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1679507624891 2023-03-22 17:53:47,011 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting dd3281f0e8f04332808f0f41f076fa4e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679507626078 2023-03-22 17:53:47,039 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#15 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:47,075 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/35e71bfb1f9349a2830bd2a4f3f80739 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/35e71bfb1f9349a2830bd2a4f3f80739 2023-03-22 17:53:47,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:47,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507687077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,096 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 35e71bfb1f9349a2830bd2a4f3f80739(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:47,096 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:47,096 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=12, startTime=1679507626755; duration=0sec 2023-03-22 17:53:47,097 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:47,278 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/c756756fe017416cbe8902994db95c1a 2023-03-22 17:53:47,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:47,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507687295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,350 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/6e9a01f31dd04d26994e22fcc12621c2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/6e9a01f31dd04d26994e22fcc12621c2 2023-03-22 17:53:47,370 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/1951dec72a244256a60278c912683606 2023-03-22 17:53:47,379 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 6e9a01f31dd04d26994e22fcc12621c2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:47,380 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:47,380 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=12, startTime=1679507626754; duration=0sec 2023-03-22 17:53:47,380 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:47,425 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/1c0f87e6a9fb4e18856396f059ee6cb0 2023-03-22 17:53:47,435 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/c756756fe017416cbe8902994db95c1a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/c756756fe017416cbe8902994db95c1a 2023-03-22 17:53:47,443 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/c756756fe017416cbe8902994db95c1a, entries=150, sequenceid=120, filesize=11.6 K 2023-03-22 17:53:47,450 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/1951dec72a244256a60278c912683606 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1951dec72a244256a60278c912683606 2023-03-22 17:53:47,469 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1951dec72a244256a60278c912683606, entries=150, sequenceid=120, filesize=11.6 K 2023-03-22 17:53:47,476 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/1c0f87e6a9fb4e18856396f059ee6cb0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1c0f87e6a9fb4e18856396f059ee6cb0 2023-03-22 17:53:47,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:47,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507687475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:47,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507687476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:47,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507687484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:47,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507687486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,493 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1c0f87e6a9fb4e18856396f059ee6cb0, entries=150, sequenceid=120, filesize=11.6 K 2023-03-22 17:53:47,495 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~374.72 KB/383712, currentSize=46.96 KB/48090 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 707ms, sequenceid=120, compaction requested=false 2023-03-22 17:53:47,496 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:47,496 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:47,498 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:47,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790ac420[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:47,502 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:47,502 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:47,505 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:47,505 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,505 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:47,507 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:47,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:47,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:47,515 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:47,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:47,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:47,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:47,515 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:47,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:47,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,520 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:47,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:47,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:47,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:47,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:47,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:47,526 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:47,527 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:47,527 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:47,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,527 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,527 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,527 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,527 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:47,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:47,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:53:47,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:47,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:47,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:47,628 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:47,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:47,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:47,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:47,653 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:47,653 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:47,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:47,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:47,655 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,655 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/de3019ca34944aa8a3bd1f1c7dab3135 2023-03-22 17:53:47,658 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:47,658 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:47,659 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,659 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:47,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:47,678 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:47,678 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:47,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:47,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:47,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:47,680 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:47,680 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:47,680 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:47,680 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:47,688 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,688 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@637152f1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:47,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:47,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,695 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:47,697 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,698 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:47,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:47,713 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:47,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:47,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:47,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:47,713 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:47,716 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,716 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:47,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,716 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,717 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:47,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:47,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:47,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:47,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:47,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:47,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,727 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:47,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:47,727 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:47,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:47,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507627663, max timestamp=1679507627715Num uniques 859; 2023-03-22 17:53:47,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:47,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507627663, max timestamp=1679507627715Num uniques 859; 2023-03-22 17:53:47,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:47,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507627663, max timestamp=1679507627715Num uniques 859; 2023-03-22 17:53:47,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:47,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:47,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507687807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,827 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:47,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:47,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:47,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:47,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:47,832 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:47,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:47,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:47,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:47,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:47,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:47,836 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,836 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:47,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:47,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:47,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:47,838 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:47,838 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:47,838 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:47,838 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:47,838 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:47,839 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:47,839 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:47,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:47,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:47,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:47,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:47,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c1dd6e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:47,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:47,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,848 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:47,849 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,849 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:47,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:47,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:47,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:47,855 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:47,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:47,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:47,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:47,855 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:47,858 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,858 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,858 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,859 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:47,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:47,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:47,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:47,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:47,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:47,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,865 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:47,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:47,866 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:47,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:47,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507687913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,966 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:47,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:47,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:47,972 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:47,972 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:47,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:47,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:47,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,975 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:47,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:47,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:47,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:47,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:47,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:47,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:47,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:47,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:47,982 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a4d357[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:47,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:47,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:47,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:47,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:47,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:47,994 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:47,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:47,995 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:47,995 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:47,995 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:47,997 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,997 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,997 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:47,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:47,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:47,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:48,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:48,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:48,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:48,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,009 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,009 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,009 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,010 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,010 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:48,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:48,010 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:48,010 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:48,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:48,110 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:48,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:48,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/d98cbb8eed514263afd67ff22df983c3 2023-03-22 17:53:48,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:48,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:48,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507688119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:48,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:48,131 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:48,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:48,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:48,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:48,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:48,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:48,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,138 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,138 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:48,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:48,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:48,149 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:48,150 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:48,150 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:48,151 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:48,151 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,154 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:48,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:48,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:48,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8202847[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:48,161 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:48,161 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,164 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:48,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:48,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:48,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/8b8a81f308744dc2aea2f817a7aca222 2023-03-22 17:53:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:48,178 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:48,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:48,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:48,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:48,179 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:48,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:48,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:48,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/de3019ca34944aa8a3bd1f1c7dab3135 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/de3019ca34944aa8a3bd1f1c7dab3135 2023-03-22 17:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:48,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:48,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:48,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:48,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:48,191 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:48,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,192 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,196 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:48,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:48,196 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:48,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:48,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/de3019ca34944aa8a3bd1f1c7dab3135, entries=200, sequenceid=134, filesize=14.0 K 2023-03-22 17:53:48,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/d98cbb8eed514263afd67ff22df983c3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/d98cbb8eed514263afd67ff22df983c3 2023-03-22 17:53:48,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/d98cbb8eed514263afd67ff22df983c3, entries=150, sequenceid=134, filesize=11.7 K 2023-03-22 17:53:48,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/8b8a81f308744dc2aea2f817a7aca222 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8b8a81f308744dc2aea2f817a7aca222 2023-03-22 17:53:48,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8b8a81f308744dc2aea2f817a7aca222, entries=150, sequenceid=134, filesize=11.7 K 2023-03-22 17:53:48,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 627ms, sequenceid=134, compaction requested=true 2023-03-22 17:53:48,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:48,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:48,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:48,230 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:48,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:48,230 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:48,233 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38236 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:48,233 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:53:48,233 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:48,233 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/6e9a01f31dd04d26994e22fcc12621c2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/c756756fe017416cbe8902994db95c1a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/de3019ca34944aa8a3bd1f1c7dab3135] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=37.3 K 2023-03-22 17:53:48,234 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35896 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:48,234 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:53:48,234 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:48,234 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/60549289b32647ad90e156f379b72366, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1951dec72a244256a60278c912683606, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/d98cbb8eed514263afd67ff22df983c3] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.1 K 2023-03-22 17:53:48,235 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 6e9a01f31dd04d26994e22fcc12621c2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679507626078 2023-03-22 17:53:48,237 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 60549289b32647ad90e156f379b72366, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679507626078 2023-03-22 17:53:48,238 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c756756fe017416cbe8902994db95c1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679507626311 2023-03-22 17:53:48,238 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1951dec72a244256a60278c912683606, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679507626311 2023-03-22 17:53:48,239 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting de3019ca34944aa8a3bd1f1c7dab3135, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507626921 2023-03-22 17:53:48,239 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d98cbb8eed514263afd67ff22df983c3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507626949 2023-03-22 17:53:48,257 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:48,259 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#22 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:48,298 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:48,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:48,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:48,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:48,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:48,308 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:48,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:48,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:48,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:48,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,311 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/1d4228a536d9472a986ebce9cc8eeedb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1d4228a536d9472a986ebce9cc8eeedb 2023-03-22 17:53:48,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:48,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:48,312 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,312 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:48,320 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/5b4b2e4d6fa348c0849dd332d5afd68b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5b4b2e4d6fa348c0849dd332d5afd68b 2023-03-22 17:53:48,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:48,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:48,329 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:48,329 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:48,329 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:48,329 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 1d4228a536d9472a986ebce9cc8eeedb(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:48,329 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:48,329 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:48,330 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=13, startTime=1679507628230; duration=0sec 2023-03-22 17:53:48,330 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:48,330 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:48,330 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:48,331 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-22 17:53:48,332 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35896 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:48,332 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:53:48,332 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:48,332 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/35e71bfb1f9349a2830bd2a4f3f80739, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1c0f87e6a9fb4e18856396f059ee6cb0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8b8a81f308744dc2aea2f817a7aca222] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.1 K 2023-03-22 17:53:48,335 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:48,335 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:48,335 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:48,335 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:48,336 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 35e71bfb1f9349a2830bd2a4f3f80739, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679507626078 2023-03-22 17:53:48,343 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:48,344 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:48,345 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1c0f87e6a9fb4e18856396f059ee6cb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679507626311 2023-03-22 17:53:48,346 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 5b4b2e4d6fa348c0849dd332d5afd68b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:48,346 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:48,346 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=13, startTime=1679507628230; duration=0sec 2023-03-22 17:53:48,346 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:48,346 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 8b8a81f308744dc2aea2f817a7aca222, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507626949 2023-03-22 17:53:48,366 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#24 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:48,400 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/bb1a1c4a87324cbfa8a0753769d9bc1a 2023-03-22 17:53:48,413 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/d49fd1b223c1407e9239bcf7569cde59 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d49fd1b223c1407e9239bcf7569cde59 2023-03-22 17:53:48,426 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into d49fd1b223c1407e9239bcf7569cde59(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:48,426 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:48,426 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=13, startTime=1679507628230; duration=0sec 2023-03-22 17:53:48,426 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:48,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:48,428 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:48,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:48,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507688460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:48,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507688490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:48,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507688491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:48,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507688492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:48,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507688493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:48,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507688564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:48,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45314 deadline: 1679507688770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,839 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/a7805e04f0824294921a32fb06e099e6 2023-03-22 17:53:48,887 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/4644b931ce0a4107a26c5baa9897d557 2023-03-22 17:53:48,904 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/bb1a1c4a87324cbfa8a0753769d9bc1a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/bb1a1c4a87324cbfa8a0753769d9bc1a 2023-03-22 17:53:48,914 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/bb1a1c4a87324cbfa8a0753769d9bc1a, entries=150, sequenceid=171, filesize=11.8 K 2023-03-22 17:53:48,915 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/a7805e04f0824294921a32fb06e099e6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a7805e04f0824294921a32fb06e099e6 2023-03-22 17:53:48,930 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a7805e04f0824294921a32fb06e099e6, entries=150, sequenceid=171, filesize=11.8 K 2023-03-22 17:53:48,933 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/4644b931ce0a4107a26c5baa9897d557 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/4644b931ce0a4107a26c5baa9897d557 2023-03-22 17:53:48,940 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/4644b931ce0a4107a26c5baa9897d557, entries=150, sequenceid=171, filesize=11.8 K 2023-03-22 17:53:48,943 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 611ms, sequenceid=171, compaction requested=false 2023-03-22 17:53:48,943 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:48,943 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:48,943 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:48,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:48,952 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:48,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:48,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@612f1568[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:48,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:48,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,962 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:48,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:48,962 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,962 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:48,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:48,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:48,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:48,965 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:48,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:48,974 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:48,974 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:48,974 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:48,974 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:48,974 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:48,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:48,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:48,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,980 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:48,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:48,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:48,985 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:48,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:48,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:48,986 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:48,986 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:48,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:48,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:48,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:48,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:48,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:48,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:49,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-22 17:53:49,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:49,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:49,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:49,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:49,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:49,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:49,089 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:49,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:49,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:49,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:49,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:49,098 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:49,098 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,105 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:49,105 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:49,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/64ba1774b43c4ad488d2458663bb87f3 2023-03-22 17:53:49,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,114 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,115 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,116 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:49,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:49,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:49,121 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,121 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:49,121 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:49,121 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:49,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c088e8e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:49,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:49,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,133 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:49,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:49,142 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:49,165 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:49,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:49,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:49,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:49,165 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:49,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:49,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/fe2959318c75483fbd8ba1036405c4f4 2023-03-22 17:53:49,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,169 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,178 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:49,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:49,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:49,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:49,180 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:49,180 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:49,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:49,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507689203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,280 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:49,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:49,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:49,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:49,287 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:49,287 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,288 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:49,288 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:49,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,291 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,292 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,292 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:49,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:49,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:49,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,294 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:49,294 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:49,294 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:49,294 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,295 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:49,295 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:49,295 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:49,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:49,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,297 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,297 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,297 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:49,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db7c633[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:49,299 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:49,299 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,301 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:49,301 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:49,301 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:49,301 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,301 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:49,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:49,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:49,308 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:49,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:49,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:49,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:49,308 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:49,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,320 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:49,325 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,326 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:49,325 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:49,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:49,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:49,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507689323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,328 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:49,328 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:49,429 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:49,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:49,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:49,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:49,435 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:49,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,437 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:49,437 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:49,438 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,438 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,439 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,440 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,440 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:49,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:49,443 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:49,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:49,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:49,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dcb9d2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:49,450 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:49,450 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,452 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:49,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,456 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:49,456 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:49,456 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:49,456 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:49,456 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:49,456 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:49,456 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:49,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:49,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:49,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:49,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:49,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:49,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,472 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,472 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:49,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:49,472 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:49,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507689528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,573 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:49,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:49,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:49,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:49,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:49,578 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:49,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:49,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:49,584 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,584 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,585 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:49,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:49,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:49,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:49,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:49,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:49,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:49,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:49,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:49,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:49,600 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:49,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc16474[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:49,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:49,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:49,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:49,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:49,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:49,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:49,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:49,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:49,610 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:49,615 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:49,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:49,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:49,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:49,615 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:49,617 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,617 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:49,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,617 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:49,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:49,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:49,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:49,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,627 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:49,627 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,627 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:49,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:49,627 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:49,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/3dff0829169344ed8b7397c2831e3a8c 2023-03-22 17:53:49,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/64ba1774b43c4ad488d2458663bb87f3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/64ba1774b43c4ad488d2458663bb87f3 2023-03-22 17:53:49,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/64ba1774b43c4ad488d2458663bb87f3, entries=150, sequenceid=185, filesize=11.8 K 2023-03-22 17:53:49,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/fe2959318c75483fbd8ba1036405c4f4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/fe2959318c75483fbd8ba1036405c4f4 2023-03-22 17:53:49,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/fe2959318c75483fbd8ba1036405c4f4, entries=150, sequenceid=185, filesize=11.8 K 2023-03-22 17:53:49,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/3dff0829169344ed8b7397c2831e3a8c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3dff0829169344ed8b7397c2831e3a8c 2023-03-22 17:53:49,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3dff0829169344ed8b7397c2831e3a8c, entries=150, sequenceid=185, filesize=11.8 K 2023-03-22 17:53:49,683 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 604ms, sequenceid=185, compaction requested=true 2023-03-22 17:53:49,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:49,683 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:49,683 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:49,683 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:49,683 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:49,683 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:49,686 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36298 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:49,686 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:53:49,686 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,686 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1d4228a536d9472a986ebce9cc8eeedb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a7805e04f0824294921a32fb06e099e6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/fe2959318c75483fbd8ba1036405c4f4] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.4 K 2023-03-22 17:53:49,687 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36298 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:49,687 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1d4228a536d9472a986ebce9cc8eeedb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507626949 2023-03-22 17:53:49,688 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:53:49,688 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,689 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5b4b2e4d6fa348c0849dd332d5afd68b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/bb1a1c4a87324cbfa8a0753769d9bc1a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/64ba1774b43c4ad488d2458663bb87f3] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.4 K 2023-03-22 17:53:49,689 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a7805e04f0824294921a32fb06e099e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679507627791 2023-03-22 17:53:49,690 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting fe2959318c75483fbd8ba1036405c4f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679507628456 2023-03-22 17:53:49,691 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5b4b2e4d6fa348c0849dd332d5afd68b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507626949 2023-03-22 17:53:49,701 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting bb1a1c4a87324cbfa8a0753769d9bc1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679507627791 2023-03-22 17:53:49,702 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 64ba1774b43c4ad488d2458663bb87f3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679507628456 2023-03-22 17:53:49,718 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:49,720 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#31 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:49,727 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:49,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:49,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:49,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:49,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:49,736 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:49,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:49,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:49,746 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,746 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,747 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:49,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:49,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:49,755 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,755 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:49,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:49,757 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/3b37a291cdf6434d8a9a1de718d062fa as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b37a291cdf6434d8a9a1de718d062fa 2023-03-22 17:53:49,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:49,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:49,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:49,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:49,762 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-22 17:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:49,763 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:49,763 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:49,763 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:49,768 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 3b37a291cdf6434d8a9a1de718d062fa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:49,768 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:49,768 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=13, startTime=1679507629683; duration=0sec 2023-03-22 17:53:49,769 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:49,769 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:49,770 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36298 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:49,770 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:53:49,770 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:49,770 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d49fd1b223c1407e9239bcf7569cde59, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/4644b931ce0a4107a26c5baa9897d557, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3dff0829169344ed8b7397c2831e3a8c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.4 K 2023-03-22 17:53:49,777 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d49fd1b223c1407e9239bcf7569cde59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507626949 2023-03-22 17:53:49,777 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4644b931ce0a4107a26c5baa9897d557, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679507627791 2023-03-22 17:53:49,778 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3dff0829169344ed8b7397c2831e3a8c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679507628456 2023-03-22 17:53:49,796 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#33 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:49,803 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/44a1b2fe804244e1991a898a5fe78c9d 2023-03-22 17:53:49,833 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/29574ce7014f41159cd82fe86fb77fa1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/29574ce7014f41159cd82fe86fb77fa1 2023-03-22 17:53:49,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:49,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:49,842 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 29574ce7014f41159cd82fe86fb77fa1(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:49,842 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:49,842 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=13, startTime=1679507629683; duration=0sec 2023-03-22 17:53:49,842 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:49,847 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/704c8ae87f6c4391acbb4265fcfc008c 2023-03-22 17:53:49,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:49,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507689879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:49,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:49,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507689984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,155 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/01f1c849cc2d4ec78c35d9647d1f2a19 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/01f1c849cc2d4ec78c35d9647d1f2a19 2023-03-22 17:53:50,165 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 01f1c849cc2d4ec78c35d9647d1f2a19(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:50,165 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:50,165 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=13, startTime=1679507629683; duration=0sec 2023-03-22 17:53:50,166 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:50,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507690192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,291 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/fc527653ed8843bd9e5767dda96150ec 2023-03-22 17:53:50,303 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/44a1b2fe804244e1991a898a5fe78c9d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/44a1b2fe804244e1991a898a5fe78c9d 2023-03-22 17:53:50,311 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/44a1b2fe804244e1991a898a5fe78c9d, entries=150, sequenceid=205, filesize=11.8 K 2023-03-22 17:53:50,314 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/704c8ae87f6c4391acbb4265fcfc008c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/704c8ae87f6c4391acbb4265fcfc008c 2023-03-22 17:53:50,321 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/704c8ae87f6c4391acbb4265fcfc008c, entries=150, sequenceid=205, filesize=11.8 K 2023-03-22 17:53:50,323 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/fc527653ed8843bd9e5767dda96150ec as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/fc527653ed8843bd9e5767dda96150ec 2023-03-22 17:53:50,339 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/fc527653ed8843bd9e5767dda96150ec, entries=150, sequenceid=205, filesize=11.8 K 2023-03-22 17:53:50,340 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 578ms, sequenceid=205, compaction requested=false 2023-03-22 17:53:50,340 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:50,341 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:50,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5515f9bc[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:50,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:50,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,349 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:50,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:50,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:50,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:50,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,354 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,354 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,354 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:50,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:50,359 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:50,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:50,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:50,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:50,359 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:50,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,374 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:50,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,385 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,386 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:50,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,386 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:50,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,486 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:50,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:50,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:50,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:50,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:50,491 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:50,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:50,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:50,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:50,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,494 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,494 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,494 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,494 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:50,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:50,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:50,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:50,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:50,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:50,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:50,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-22 17:53:50,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:50,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:50,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:50,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:50,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:50,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:50,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:50,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6402b73[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:50,508 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:50,508 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,511 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,511 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:50,516 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:50,520 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:50,521 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:50,521 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:50,521 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:50,521 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:50,523 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,523 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,523 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,525 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,527 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,534 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,534 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,534 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,534 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,534 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,535 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,535 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,535 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,535 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,535 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:50,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:50,535 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,535 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:50,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,535 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507690544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507690545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507690546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507690550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507690565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,636 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:50,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:50,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:50,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:50,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:50,641 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:50,641 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:50,641 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:50,641 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:50,642 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,642 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,645 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,645 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:50,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:50,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:50,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:50,654 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,655 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:50,655 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:50,655 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:50,655 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:50,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507690656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,658 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,658 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507690658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507690658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507690661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:50,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d30040[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:50,663 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:50,663 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,665 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:50,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:50,666 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,666 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:50,672 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:50,679 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:50,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:50,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:50,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:50,679 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:50,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,682 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507690678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,685 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:50,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:50,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,692 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,692 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:50,692 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:50,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:50,792 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:50,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:50,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:50,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:50,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:50,807 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:50,807 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:50,807 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:50,807 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:50,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,809 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:50,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:50,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:50,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:50,813 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,813 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:50,813 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:50,813 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:50,823 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,826 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,826 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:50,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c7260a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:50,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:50,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:50,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:50,830 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:50,835 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:50,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:50,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:50,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:50,835 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:50,837 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,837 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,837 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,838 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,840 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,843 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:50,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:50,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:50,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,844 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,844 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:50,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,844 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,844 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:50,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507690860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507690864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507690865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507690865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:50,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507690886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,945 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:50,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:50,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:50,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:50,949 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:50,949 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:50,949 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:50,949 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:50,949 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,949 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,951 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,951 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,951 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:50,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:50,953 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,953 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:50,953 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:50,953 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:50,955 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,956 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,956 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:50,956 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:50,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@be5f997[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:50,958 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:50,959 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,960 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,960 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:50,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:50,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:50,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:50,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:50,967 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:50,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:50,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:50,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:50,967 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:50,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/f11e0d5a83e0441c972b46b4c3545dd9 2023-03-22 17:53:50,971 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:50,971 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,971 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:50,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,992 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,992 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:50,992 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:50,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:50,992 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:50,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:50,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:50,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:50,992 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:50,993 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:50,993 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,993 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:50,993 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:50,993 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:50,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:50,993 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,093 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:51,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:51,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:51,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:51,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:51,098 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:51,098 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,098 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:51,098 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:51,099 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,099 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,102 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:51,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:51,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:51,104 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:51,104 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,104 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:51,104 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:51,104 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:51,105 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:51,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,107 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c79502b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:51,109 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:51,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,112 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:51,112 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:51,112 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:51,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:51,114 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,114 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:51,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:51,117 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:51,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:51,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:51,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:51,117 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:51,120 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,120 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,120 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,123 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,125 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,125 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,126 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:51,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:51,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,126 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:51,126 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:51,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507691164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507691165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507691167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507691167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507691190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,227 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:51,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:51,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:51,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:51,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:51,231 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:51,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:51,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:51,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,233 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:51,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:51,235 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,235 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:51,235 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:51,235 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:51,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@133790e4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:51,239 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:51,240 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,246 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:51,247 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,247 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:51,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:51,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:51,251 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:51,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:51,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:51,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:51,251 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:51,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,255 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,256 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:51,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,261 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:51,261 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:51,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,362 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:51,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:51,366 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:51,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:51,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:51,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,367 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:51,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:51,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:51,371 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,371 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:51,371 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:51,371 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:51,379 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:51,381 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:51,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0813fe[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:51,381 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:51,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:51,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:51,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/3b0f183c8f0746449e167bb0e1226ea0 2023-03-22 17:53:51,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:51,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:51,390 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:51,390 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:51,390 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:51,390 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:51,390 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:51,393 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,393 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:51,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,403 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:51,404 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:51,504 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:51,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:51,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:51,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:51,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:51,509 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:51,509 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,509 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:51,509 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:51,510 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,510 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,511 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:51,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:51,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:51,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:51,522 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,522 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:51,523 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:51,523 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:51,528 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f056049[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:51,530 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:51,530 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,531 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,532 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:51,532 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:51,532 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:51,534 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:51,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:51,537 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:51,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:51,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:51,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:51,537 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:51,539 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,539 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,539 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,539 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,541 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:51,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:51,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:51,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,545 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:51,545 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:51,645 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:51,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:51,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:51,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:51,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:51,668 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:51,668 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,668 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:51,668 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:51,668 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,674 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,674 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:51,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:51,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,675 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,676 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:51,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507691670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,676 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:51,676 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:51,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507691671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:51,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507691673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,678 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507691676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39bb8f8a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:51,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:51,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,683 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:51,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:51,688 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:51,697 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:51,697 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:51,698 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:51,698 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:51,698 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:51,700 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,700 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,700 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,701 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:51,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507691698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,704 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:51,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:51,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,711 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:51,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:51,711 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:51,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,801 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/d208ef6d2ed0403abc6276536c941a03 2023-03-22 17:53:51,811 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:51,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/f11e0d5a83e0441c972b46b4c3545dd9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f11e0d5a83e0441c972b46b4c3545dd9 2023-03-22 17:53:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:51,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:51,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f11e0d5a83e0441c972b46b4c3545dd9, entries=150, sequenceid=221, filesize=11.8 K 2023-03-22 17:53:51,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:51,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:51,827 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:51,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:51,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:51,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/3b0f183c8f0746449e167bb0e1226ea0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b0f183c8f0746449e167bb0e1226ea0 2023-03-22 17:53:51,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:51,843 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b0f183c8f0746449e167bb0e1226ea0, entries=150, sequenceid=221, filesize=11.8 K 2023-03-22 17:53:51,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/d208ef6d2ed0403abc6276536c941a03 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d208ef6d2ed0403abc6276536c941a03 2023-03-22 17:53:51,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:51,852 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:51,852 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:51,852 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:51,863 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:51,863 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:51,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d208ef6d2ed0403abc6276536c941a03, entries=150, sequenceid=221, filesize=11.8 K 2023-03-22 17:53:51,869 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,870 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:51,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 1370ms, sequenceid=221, compaction requested=true 2023-03-22 17:53:51,870 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:51,871 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,871 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:51,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:51,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:51,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:51,871 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:51,871 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:51,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:51,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:51,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:51,871 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:51,873 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:51,873 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:53:51,873 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,873 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,873 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/01f1c849cc2d4ec78c35d9647d1f2a19, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/44a1b2fe804244e1991a898a5fe78c9d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f11e0d5a83e0441c972b46b4c3545dd9] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.6 K 2023-03-22 17:53:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,873 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:51,877 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:53:51,877 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,877 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b37a291cdf6434d8a9a1de718d062fa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/704c8ae87f6c4391acbb4265fcfc008c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b0f183c8f0746449e167bb0e1226ea0] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.6 K 2023-03-22 17:53:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,883 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 01f1c849cc2d4ec78c35d9647d1f2a19, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679507628456 2023-03-22 17:53:51,892 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3b37a291cdf6434d8a9a1de718d062fa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679507628456 2023-03-22 17:53:51,893 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 44a1b2fe804244e1991a898a5fe78c9d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507629188 2023-03-22 17:53:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,893 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 704c8ae87f6c4391acbb4265fcfc008c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507629188 2023-03-22 17:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,893 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3b0f183c8f0746449e167bb0e1226ea0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679507629857 2023-03-22 17:53:51,893 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f11e0d5a83e0441c972b46b4c3545dd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679507629857 2023-03-22 17:53:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:51,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a498e9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:51,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:51,898 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,899 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:51,900 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:51,900 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:51,900 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,900 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:51,902 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:51,902 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,902 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:51,907 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:51,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:51,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:51,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:51,907 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:51,911 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,911 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:51,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,911 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,916 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,919 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:51,919 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:51,919 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:51,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:51,919 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:51,919 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:51,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:51,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:51,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:51,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:51,920 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:51,920 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:51,925 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:51,925 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:51,951 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/59fa985310e545918e1d4514de8d8dc6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/59fa985310e545918e1d4514de8d8dc6 2023-03-22 17:53:51,960 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 59fa985310e545918e1d4514de8d8dc6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:51,960 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:51,960 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=13, startTime=1679507631871; duration=0sec 2023-03-22 17:53:51,960 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:51,960 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:51,963 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:51,963 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:53:51,963 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:51,963 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/29574ce7014f41159cd82fe86fb77fa1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/fc527653ed8843bd9e5767dda96150ec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d208ef6d2ed0403abc6276536c941a03] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.6 K 2023-03-22 17:53:51,964 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 29574ce7014f41159cd82fe86fb77fa1, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679507628456 2023-03-22 17:53:51,965 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting fc527653ed8843bd9e5767dda96150ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507629188 2023-03-22 17:53:51,965 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d208ef6d2ed0403abc6276536c941a03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679507629857 2023-03-22 17:53:51,974 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#41 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:51,993 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/ad4ac79fc2a84527a832c4c91b7720eb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ad4ac79fc2a84527a832c4c91b7720eb 2023-03-22 17:53:52,007 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into ad4ac79fc2a84527a832c4c91b7720eb(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:52,007 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:52,007 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=13, startTime=1679507631871; duration=0sec 2023-03-22 17:53:52,008 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:52,020 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:52,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:52,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:52,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:52,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:52,032 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:52,032 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:52,032 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:52,032 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:52,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:52,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:52,034 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:52,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:52,034 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:52,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:52,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:52,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:52,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:52,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:52,036 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:52,036 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:52,037 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-22 17:53:52,037 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:52,037 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:52,037 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:52,037 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:52,037 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:52,037 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:52,053 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/19cea5577f47437d94040440c70fd427 2023-03-22 17:53:52,354 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/0be1d79c5785449b8c9be4321e859989 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/0be1d79c5785449b8c9be4321e859989 2023-03-22 17:53:52,363 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 0be1d79c5785449b8c9be4321e859989(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:52,363 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:52,364 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=13, startTime=1679507631871; duration=0sec 2023-03-22 17:53:52,364 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:52,471 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/1c34e5831e8c4430873fd81e900f9153 2023-03-22 17:53:52,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:52,680 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:52,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507692704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507692705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507692710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507692719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507692719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507692823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507692824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507692822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:52,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507692825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,943 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/c99dde605dcc42c7a81893157d79301a 2023-03-22 17:53:52,954 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/19cea5577f47437d94040440c70fd427 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/19cea5577f47437d94040440c70fd427 2023-03-22 17:53:52,961 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/19cea5577f47437d94040440c70fd427, entries=150, sequenceid=242, filesize=11.8 K 2023-03-22 17:53:52,962 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/1c34e5831e8c4430873fd81e900f9153 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1c34e5831e8c4430873fd81e900f9153 2023-03-22 17:53:52,968 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1c34e5831e8c4430873fd81e900f9153, entries=150, sequenceid=242, filesize=11.8 K 2023-03-22 17:53:52,971 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/c99dde605dcc42c7a81893157d79301a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/c99dde605dcc42c7a81893157d79301a 2023-03-22 17:53:52,978 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/c99dde605dcc42c7a81893157d79301a, entries=150, sequenceid=242, filesize=11.8 K 2023-03-22 17:53:52,979 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 942ms, sequenceid=242, compaction requested=false 2023-03-22 17:53:52,980 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:52,980 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:52,980 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:52,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:52,983 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:52,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:52,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:52,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:52,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da8ee20[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:52,989 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:52,989 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:52,993 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:52,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:52,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:52,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:52,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:53,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:53,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:53,013 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:53,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:53,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:53,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:53,013 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:53,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:53,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:53,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:53,018 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:53,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:53,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:53,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,030 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:53,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:53,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:53,031 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-22 17:53:53,031 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:53,031 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,031 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:53,031 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:53,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:53,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:53,031 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:53,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:53,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/daacd7cba00745a194e17c9ed13e2345 2023-03-22 17:53:53,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507693053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507693054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507693056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507693056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,078 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/e9407018377147ebbfde5a97ab140e55 2023-03-22 17:53:53,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/076c51e348f04ff1b529203dbace9d4a 2023-03-22 17:53:53,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/daacd7cba00745a194e17c9ed13e2345 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/daacd7cba00745a194e17c9ed13e2345 2023-03-22 17:53:53,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/daacd7cba00745a194e17c9ed13e2345, entries=150, sequenceid=258, filesize=11.8 K 2023-03-22 17:53:53,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/e9407018377147ebbfde5a97ab140e55 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e9407018377147ebbfde5a97ab140e55 2023-03-22 17:53:53,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e9407018377147ebbfde5a97ab140e55, entries=150, sequenceid=258, filesize=11.8 K 2023-03-22 17:53:53,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/076c51e348f04ff1b529203dbace9d4a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/076c51e348f04ff1b529203dbace9d4a 2023-03-22 17:53:53,132 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:53,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:53,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:53,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:53,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:53,136 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:53,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:53,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:53,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:53,136 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/076c51e348f04ff1b529203dbace9d4a, entries=150, sequenceid=258, filesize=11.8 K 2023-03-22 17:53:53,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,137 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 106ms, sequenceid=258, compaction requested=true 2023-03-22 17:53:53,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:53,138 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:53,138 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:53,138 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:53,138 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:53,138 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:53,140 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:53,140 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:53,140 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36602 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:53,140 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:53:53,140 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,140 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36602 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:53,140 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/0be1d79c5785449b8c9be4321e859989, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1c34e5831e8c4430873fd81e900f9153, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e9407018377147ebbfde5a97ab140e55] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.7 K 2023-03-22 17:53:53,141 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:53:53,141 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,141 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/59fa985310e545918e1d4514de8d8dc6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/19cea5577f47437d94040440c70fd427, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/daacd7cba00745a194e17c9ed13e2345] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.7 K 2023-03-22 17:53:53,141 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0be1d79c5785449b8c9be4321e859989, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679507629857 2023-03-22 17:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,156 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 59fa985310e545918e1d4514de8d8dc6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679507629857 2023-03-22 17:53:53,157 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1c34e5831e8c4430873fd81e900f9153, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679507630541 2023-03-22 17:53:53,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:53,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:53,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:53,161 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-22 17:53:53,161 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,161 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:53,161 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e9407018377147ebbfde5a97ab140e55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679507632702 2023-03-22 17:53:53,162 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 19cea5577f47437d94040440c70fd427, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679507630541 2023-03-22 17:53:53,169 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting daacd7cba00745a194e17c9ed13e2345, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679507632702 2023-03-22 17:53:53,176 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:53,181 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#49 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:53,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:53,185 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:53,186 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507693212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507693216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507693219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507693221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,246 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/08dad7ee6d7a4716babb2802e6761314 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/08dad7ee6d7a4716babb2802e6761314 2023-03-22 17:53:53,260 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 08dad7ee6d7a4716babb2802e6761314(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:53,260 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:53,260 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=13, startTime=1679507633138; duration=0sec 2023-03-22 17:53:53,260 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:53,260 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:53,263 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36602 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:53,263 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:53:53,264 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,264 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ad4ac79fc2a84527a832c4c91b7720eb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/c99dde605dcc42c7a81893157d79301a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/076c51e348f04ff1b529203dbace9d4a] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=35.7 K 2023-03-22 17:53:53,265 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting ad4ac79fc2a84527a832c4c91b7720eb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679507629857 2023-03-22 17:53:53,266 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c99dde605dcc42c7a81893157d79301a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679507630541 2023-03-22 17:53:53,269 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 076c51e348f04ff1b529203dbace9d4a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679507632702 2023-03-22 17:53:53,273 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/ccfaccbe0d754795bc0dab6f06086bc8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/ccfaccbe0d754795bc0dab6f06086bc8 2023-03-22 17:53:53,277 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/237efc1e44934537aa20fd641db32ddc 2023-03-22 17:53:53,284 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into ccfaccbe0d754795bc0dab6f06086bc8(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:53,285 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:53,285 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=13, startTime=1679507633138; duration=0sec 2023-03-22 17:53:53,285 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:53,289 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#51 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:53,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507693322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507693325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507693325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507693326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507693528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507693528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507693529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507693534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,714 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/b3cbf39a4a974dd894aac654e161c0a2 2023-03-22 17:53:53,756 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/ed5d50ad6ec740129a5172652e197edc as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ed5d50ad6ec740129a5172652e197edc 2023-03-22 17:53:53,764 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into ed5d50ad6ec740129a5172652e197edc(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:53,764 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:53,764 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=13, startTime=1679507633138; duration=0sec 2023-03-22 17:53:53,765 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:53,779 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/2b09f8a3a83248e2b259551f8bc851e1 2023-03-22 17:53:53,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/237efc1e44934537aa20fd641db32ddc as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/237efc1e44934537aa20fd641db32ddc 2023-03-22 17:53:53,793 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/237efc1e44934537aa20fd641db32ddc, entries=150, sequenceid=277, filesize=11.9 K 2023-03-22 17:53:53,794 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/b3cbf39a4a974dd894aac654e161c0a2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/b3cbf39a4a974dd894aac654e161c0a2 2023-03-22 17:53:53,801 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/b3cbf39a4a974dd894aac654e161c0a2, entries=150, sequenceid=277, filesize=11.9 K 2023-03-22 17:53:53,802 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/2b09f8a3a83248e2b259551f8bc851e1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/2b09f8a3a83248e2b259551f8bc851e1 2023-03-22 17:53:53,807 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/2b09f8a3a83248e2b259551f8bc851e1, entries=150, sequenceid=277, filesize=11.9 K 2023-03-22 17:53:53,809 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 648ms, sequenceid=277, compaction requested=false 2023-03-22 17:53:53,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:53,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:53,813 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:53,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@597bd0f4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:53,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:53,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:53,821 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,821 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:53,824 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:53,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:53,831 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:53,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:53,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:53,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:53,831 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:53,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:53,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:53,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:53,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,838 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,839 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-22 17:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:53,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:53,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:53,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:53,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:53,846 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:53,846 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:53,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:53,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:53,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507693869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/35337279be4044118e1a6be6a4f27b14 2023-03-22 17:53:53,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507693871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507693871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:53,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507693871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,896 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/4e3a317e29694e39b5ba4253942358a6 2023-03-22 17:53:53,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/6cd11b2f3a1e434f922f1c2057a63801 2023-03-22 17:53:53,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/35337279be4044118e1a6be6a4f27b14 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/35337279be4044118e1a6be6a4f27b14 2023-03-22 17:53:53,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/35337279be4044118e1a6be6a4f27b14, entries=200, sequenceid=295, filesize=14.3 K 2023-03-22 17:53:53,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/4e3a317e29694e39b5ba4253942358a6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4e3a317e29694e39b5ba4253942358a6 2023-03-22 17:53:53,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4e3a317e29694e39b5ba4253942358a6, entries=150, sequenceid=295, filesize=11.9 K 2023-03-22 17:53:53,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/6cd11b2f3a1e434f922f1c2057a63801 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/6cd11b2f3a1e434f922f1c2057a63801 2023-03-22 17:53:53,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/6cd11b2f3a1e434f922f1c2057a63801, entries=150, sequenceid=295, filesize=11.9 K 2023-03-22 17:53:53,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 100ms, sequenceid=295, compaction requested=true 2023-03-22 17:53:53,941 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:53,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:53,941 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:53,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:53,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:53,941 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:53,942 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:53,942 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:53:53,942 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37004 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:53,942 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,942 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:53:53,942 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,942 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/ccfaccbe0d754795bc0dab6f06086bc8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/237efc1e44934537aa20fd641db32ddc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/35337279be4044118e1a6be6a4f27b14] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=38.5 K 2023-03-22 17:53:53,942 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/08dad7ee6d7a4716babb2802e6761314, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/b3cbf39a4a974dd894aac654e161c0a2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4e3a317e29694e39b5ba4253942358a6] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=36.1 K 2023-03-22 17:53:53,943 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting ccfaccbe0d754795bc0dab6f06086bc8, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679507632702 2023-03-22 17:53:53,943 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 08dad7ee6d7a4716babb2802e6761314, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679507632702 2023-03-22 17:53:53,944 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 237efc1e44934537aa20fd641db32ddc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679507633054 2023-03-22 17:53:53,944 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b3cbf39a4a974dd894aac654e161c0a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679507633054 2023-03-22 17:53:53,944 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 35337279be4044118e1a6be6a4f27b14, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679507633213 2023-03-22 17:53:53,945 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4e3a317e29694e39b5ba4253942358a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679507633213 2023-03-22 17:53:53,946 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:53,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:53,968 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:53,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:53,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:53,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:53,970 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,970 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,973 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:53,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:53,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:53,974 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:53,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:53,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:53,978 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#58 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:53,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:53,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-22 17:53:53,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:53,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507633858, max timestamp=1679507633981Num uniques 859; 2023-03-22 17:53:53,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:53,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507633858, max timestamp=1679507633980Num uniques 859; 2023-03-22 17:53:53,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507633858, max timestamp=1679507633981Num uniques 859; 2023-03-22 17:53:53,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:53,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:53,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:53,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:53,987 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:53,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:53,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:53,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:53,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:53,990 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,990 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:53,991 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:53,991 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,994 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cd9c38e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:53,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:53,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,999 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:53,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:53,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:54,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:54,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:54,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:54,002 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:54,002 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,002 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,002 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:54,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:54,005 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:54,005 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:54,005 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:54,005 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:54,005 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:54,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,009 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:54,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:54,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:54,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,017 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:54,017 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,024 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/5830f3124e0444d48887c06aa2723048 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5830f3124e0444d48887c06aa2723048 2023-03-22 17:53:54,032 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 5830f3124e0444d48887c06aa2723048(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:54,033 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:54,033 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=13, startTime=1679507633941; duration=0sec 2023-03-22 17:53:54,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507634013, max timestamp=1679507634026Num uniques 859; 2023-03-22 17:53:54,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:54,033 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:54,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507634013, max timestamp=1679507634027Num uniques 859; 2023-03-22 17:53:54,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507634013, max timestamp=1679507634027Num uniques 859; 2023-03-22 17:53:54,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:54,034 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:54,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:53:54,037 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37004 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:54,037 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:53:54,038 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,038 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ed5d50ad6ec740129a5172652e197edc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/2b09f8a3a83248e2b259551f8bc851e1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/6cd11b2f3a1e434f922f1c2057a63801] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=36.1 K 2023-03-22 17:53:54,038 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting ed5d50ad6ec740129a5172652e197edc, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679507632702 2023-03-22 17:53:54,039 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2b09f8a3a83248e2b259551f8bc851e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679507633054 2023-03-22 17:53:54,041 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 6cd11b2f3a1e434f922f1c2057a63801, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679507633213 2023-03-22 17:53:54,047 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/429b55bde83a4741864c75e7cbc6126b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/429b55bde83a4741864c75e7cbc6126b 2023-03-22 17:53:54,054 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 429b55bde83a4741864c75e7cbc6126b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:54,054 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:54,054 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=13, startTime=1679507633941; duration=0sec 2023-03-22 17:53:54,055 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:54,061 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#60 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:54,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679507634058, max timestamp=1679507634068Num uniques 1018; 2023-03-22 17:53:54,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-22 17:53:54,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679507634058, max timestamp=1679507634068Num uniques 1018; 2023-03-22 17:53:54,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-22 17:53:54,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679507634058, max timestamp=1679507634068Num uniques 1018; 2023-03-22 17:53:54,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-22 17:53:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507694089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507694092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507694098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507694099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,117 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:54,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:54,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:54,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:54,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:54,148 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:54,148 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,148 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:54,148 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:54,149 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,149 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,150 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:54,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:54,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:54,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:54,160 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,160 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:54,160 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:54,160 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,163 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,163 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:54,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e88772c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:54,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:54,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,168 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:54,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,169 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:54,175 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:54,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:54,180 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:54,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:54,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:54,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:54,180 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:54,183 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,183 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,183 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,188 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,188 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,188 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,188 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:54,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:54,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,188 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,189 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:54,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,189 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:54,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507694200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507694200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507694200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507694206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,289 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:54,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:54,293 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:54,293 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,293 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:54,293 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:54,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,302 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,303 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,303 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:54,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:54,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:54,305 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,305 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:54,305 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:54,305 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:54,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,308 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:54,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449b4133[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:54,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:54,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:54,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:54,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:54,315 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:54,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:54,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:54,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:54,315 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:54,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,330 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,331 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,334 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,334 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,334 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,334 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:54,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:54,334 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,335 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:54,335 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:54,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507694405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507694408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507694409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507694409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,435 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:54,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:54,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:54,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:54,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:54,439 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:54,439 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,439 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:54,439 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:54,439 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,439 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,441 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:54,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:54,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:54,443 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,443 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:54,443 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:54,443 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:54,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/dea26a033b99477db4dbac7b72c28d3f 2023-03-22 17:53:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:54,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43e0d524[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:54,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:54,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,456 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:54,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:54,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:54,469 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:54,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:54,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:54,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:54,469 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:54,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,477 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,481 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:54,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:54,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:54,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:54,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:54,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/9abcfc68cb2e4dc1984e9562ec737910 2023-03-22 17:53:54,491 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/3bcbfbea38e0489aa19505f818647347 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3bcbfbea38e0489aa19505f818647347 2023-03-22 17:53:54,499 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 3bcbfbea38e0489aa19505f818647347(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:54,499 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:54,499 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=13, startTime=1679507633941; duration=0sec 2023-03-22 17:53:54,499 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:54,582 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:54,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:54,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:54,591 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:54,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:54,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:54,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,593 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:54,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:54,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:54,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:54,596 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,596 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:54,596 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:54,596 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:54,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:54,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:54,598 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:54,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31d7ab65[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:54,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:54,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:54,604 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:54,608 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:54,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:54,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:54,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:54,608 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:54,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:54,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:54,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:54,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,623 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:54,623 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:54,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507694708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507694711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507694712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507694712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,723 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:54,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:54,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:54,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:54,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507694727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,731 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:53:54,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:54,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:54,732 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:54,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:54,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:54,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,734 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,734 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,734 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:54,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:54,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:54,736 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,736 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:54,736 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:54,737 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:54,738 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cab9b64[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:54,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:54,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:54,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:54,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:54,748 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:54,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:54,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:54,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:54,748 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:54,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:54,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:54,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,760 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:54,760 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:54,860 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:54,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:54,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:54,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:54,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:54,864 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:54,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:54,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:54,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,865 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,866 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:54,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:54,871 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,871 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:54,872 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:54,872 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:54,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6227cf1b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:54,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:54,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,878 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:54,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:54,880 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:54,882 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:54,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:54,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:54,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:54,882 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:54,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:54,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:54,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,889 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,889 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:54,889 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:54,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:54,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:54,889 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:54,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:54,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:54,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:54,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:54,896 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:54,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,896 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:54,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:54,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:54,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:54,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/a60076075f59461ab05036dabb15377b 2023-03-22 17:53:54,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/dea26a033b99477db4dbac7b72c28d3f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/dea26a033b99477db4dbac7b72c28d3f 2023-03-22 17:53:54,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/dea26a033b99477db4dbac7b72c28d3f, entries=150, sequenceid=317, filesize=11.9 K 2023-03-22 17:53:54,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/9abcfc68cb2e4dc1984e9562ec737910 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/9abcfc68cb2e4dc1984e9562ec737910 2023-03-22 17:53:54,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/9abcfc68cb2e4dc1984e9562ec737910, entries=150, sequenceid=317, filesize=11.9 K 2023-03-22 17:53:54,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/a60076075f59461ab05036dabb15377b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a60076075f59461ab05036dabb15377b 2023-03-22 17:53:54,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a60076075f59461ab05036dabb15377b, entries=150, sequenceid=317, filesize=11.9 K 2023-03-22 17:53:54,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 971ms, sequenceid=317, compaction requested=false 2023-03-22 17:53:54,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:54,996 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:54,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:54,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:54,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:55,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:55,000 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:55,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:55,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:55,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:55,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:55,002 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:55,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:55,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:55,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:55,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:55,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:55,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:55,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:55,003 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:55,004 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=154.31 KB heapSize=429.84 KB 2023-03-22 17:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:55,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:55,218 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:55,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507695220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507695221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507695221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507695222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507695323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507695325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507695325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507695325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,419 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/54f6552cb91d444080768d0940a1f1a7 2023-03-22 17:53:55,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507695525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507695529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507695531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507695531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45324 deadline: 1679507695828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,833 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/f1171554ade644d48d384e9cfd3143ef 2023-03-22 17:53:55,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45348 deadline: 1679507695832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507695834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:55,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507695836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,849 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/feedb777e5934374be208d698af281d6 2023-03-22 17:53:55,854 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/54f6552cb91d444080768d0940a1f1a7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/54f6552cb91d444080768d0940a1f1a7 2023-03-22 17:53:55,860 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/54f6552cb91d444080768d0940a1f1a7, entries=150, sequenceid=374, filesize=11.9 K 2023-03-22 17:53:55,861 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/f1171554ade644d48d384e9cfd3143ef as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f1171554ade644d48d384e9cfd3143ef 2023-03-22 17:53:55,866 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f1171554ade644d48d384e9cfd3143ef, entries=150, sequenceid=374, filesize=11.9 K 2023-03-22 17:53:55,867 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/feedb777e5934374be208d698af281d6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/feedb777e5934374be208d698af281d6 2023-03-22 17:53:55,873 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/feedb777e5934374be208d698af281d6, entries=150, sequenceid=374, filesize=11.9 K 2023-03-22 17:53:55,874 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.80 KB/440112, currentSize=33.54 KB/34350 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 870ms, sequenceid=374, compaction requested=true 2023-03-22 17:53:55,874 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:55,875 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:55,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:55,877 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:55,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:55,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272b6f8e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:55,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:55,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:55,881 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:55,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:55,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:55,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:55,882 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:55,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:55,885 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:55,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:55,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:55,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:55,886 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:55,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:55,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,888 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:55,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:55,889 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:55,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:55,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:55,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:55,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:55,897 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:55,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:55,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:55,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:55,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:55,897 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:55,897 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:55,897 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:55,897 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:55,897 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:55,897 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:55,997 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:55,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:56,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:56,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:56,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:56,001 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:56,001 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:56,001 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:56,001 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:56,001 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,001 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:56,003 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:56,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:56,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:56,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:56,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:56,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:56,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:56,005 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:56,005 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:56,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:56,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:56,005 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-22 17:53:56,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:56,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:56,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:56,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:56,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:56,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:56,024 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/a581c2356e6a410e9be965a1d9a7fd29 2023-03-22 17:53:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:56,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:56,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507636371, max timestamp=1679507636385Num uniques 859; 2023-03-22 17:53:56,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:53:56,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507636371, max timestamp=1679507636385Num uniques 859; 2023-03-22 17:53:56,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507636371, max timestamp=1679507636385Num uniques 859; 2023-03-22 17:53:56,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:53:56,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:53:56,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507696417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507696421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507696423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507696425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,453 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/e17b86b27c394df8b12c2054936c09db 2023-03-22 17:53:56,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507696527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507696530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507696530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45350 deadline: 1679507696531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507696733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507696733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507696734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:56,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507696735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,878 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/0f6f6103adb54c85a74bffb5a5938065 2023-03-22 17:53:56,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/a581c2356e6a410e9be965a1d9a7fd29 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a581c2356e6a410e9be965a1d9a7fd29 2023-03-22 17:53:56,889 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a581c2356e6a410e9be965a1d9a7fd29, entries=150, sequenceid=382, filesize=11.9 K 2023-03-22 17:53:56,890 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/e17b86b27c394df8b12c2054936c09db as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e17b86b27c394df8b12c2054936c09db 2023-03-22 17:53:56,900 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e17b86b27c394df8b12c2054936c09db, entries=150, sequenceid=382, filesize=11.9 K 2023-03-22 17:53:56,903 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/0f6f6103adb54c85a74bffb5a5938065 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/0f6f6103adb54c85a74bffb5a5938065 2023-03-22 17:53:56,910 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/0f6f6103adb54c85a74bffb5a5938065, entries=150, sequenceid=382, filesize=11.9 K 2023-03-22 17:53:56,911 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 906ms, sequenceid=382, compaction requested=true 2023-03-22 17:53:56,911 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:56,911 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:56,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:56,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:56,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:56,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bab748d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:56,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:56,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:56,919 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:56,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:56,919 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:56,919 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:56,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:56,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,921 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:56,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:56,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:56,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:56,925 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:56,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:56,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:56,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:56,925 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:56,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:56,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:56,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:56,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:56,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:56,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:56,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:56,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:56,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:56,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:56,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:56,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:56,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:56,936 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:56,936 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:56,938 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:56,938 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:56,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:56,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,036 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:57,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:57,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:57,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=161.02 KB heapSize=451.31 KB 2023-03-22 17:53:57,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:57,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:53:57,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:57,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:53:57,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:57,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:53:57,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:57,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:57,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:57,042 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:57,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:57,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:57,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,045 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,045 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,045 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,045 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:57,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:57,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:57,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:57,046 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:57,046 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,047 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:57,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507697042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507697043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507697044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507697044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:57,048 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,048 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:57,048 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:57,048 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:57,053 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b7baa27[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:57,055 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:57,055 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,056 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:57,057 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,057 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:57,060 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:57,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:57,062 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:57,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:57,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:57,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:57,062 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:57,064 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,064 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,064 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:57,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:57,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,068 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:57,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:57,069 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:57,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507697148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507697148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507697149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,169 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:57,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:57,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:57,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:57,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:57,173 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:57,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:57,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:57,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,175 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,176 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,176 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:57,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:57,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:57,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:57,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:57,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:57,178 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,178 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:57,178 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:57,178 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:57,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:57,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f8e344[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:57,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:57,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:57,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:57,186 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:57,190 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:57,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:57,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:57,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:57,190 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:57,192 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,192 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,192 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,201 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:57,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:57,202 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:57,202 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:57,302 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:57,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:57,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:57,309 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:57,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:57,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:57,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,314 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:57,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:57,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:57,320 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b53b12a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:57,325 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:57,325 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,326 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:57,327 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:57,327 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,327 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:57,332 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:57,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:57,335 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:57,335 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:57,335 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:57,335 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:57,335 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:57,337 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,338 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,338 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,338 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,345 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,345 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:57,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,346 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:57,346 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:57,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,346 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:57,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507697350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507697351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507697354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,446 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:57,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:57,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:57,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:57,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:57,449 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:57,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:57,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:57,450 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,450 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,452 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,453 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,453 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,453 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:57,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/943be75650f848bea8329a2ad27f85ed 2023-03-22 17:53:57,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:57,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:57,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:57,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:57,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:57,457 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:57,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5baca7d1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:57,462 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:57,462 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,463 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:57,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:57,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:57,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:57,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:57,466 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:57,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:57,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:57,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:57,466 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:57,469 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,469 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,469 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/93605b870d0044398df72fe9702fe8dd 2023-03-22 17:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,474 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,474 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,474 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,474 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:57,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:57,474 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:57,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,475 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:57,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:57,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507697549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:57,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:57,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:57,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:57,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:57,578 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:57,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:57,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:57,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,580 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,580 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:57,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:57,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:57,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:57,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:57,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:57,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:57,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:57,582 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,582 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:57,582 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:57,582 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:57,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:57,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:57,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:57,584 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:57,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d500411[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:57,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:57,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,589 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,589 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:57,589 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:57,589 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:57,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,590 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:57,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:57,593 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:57,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:57,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:57,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:57,593 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:57,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:57,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,601 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:57,601 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:57,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507697653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507697654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:57,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507697656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,701 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:57,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:57,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:57,704 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:57,704 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,704 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:57,704 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:57,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,709 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,709 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:57,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:57,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:57,711 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,711 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:57,711 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:57,711 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:57,714 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:57,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da2243c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:57,716 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:57,717 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,720 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:57,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,722 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:57,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:57,725 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:57,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:57,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:57,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:57,725 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:57,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:57,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:57,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:57,739 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:57,739 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:57,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,839 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:57,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:57,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:57,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:57,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:57,843 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:57,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:57,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:57,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:57,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:57,848 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,848 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:57,848 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:57,848 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:57,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bce1c43[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:57,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:57,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,856 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:57,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:57,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:57,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:57,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:57,861 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:57,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:57,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:57,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:57,861 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:57,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:57,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:57,865 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,870 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:57,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:57,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:57,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:57,871 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:57,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:57,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:57,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/ac91e00f0c5749d8ba63eb2eb9abea66 2023-03-22 17:53:57,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/943be75650f848bea8329a2ad27f85ed as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/943be75650f848bea8329a2ad27f85ed 2023-03-22 17:53:57,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/943be75650f848bea8329a2ad27f85ed, entries=200, sequenceid=420, filesize=14.3 K 2023-03-22 17:53:57,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/93605b870d0044398df72fe9702fe8dd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/93605b870d0044398df72fe9702fe8dd 2023-03-22 17:53:57,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/93605b870d0044398df72fe9702fe8dd, entries=150, sequenceid=420, filesize=11.9 K 2023-03-22 17:53:57,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/ac91e00f0c5749d8ba63eb2eb9abea66 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ac91e00f0c5749d8ba63eb2eb9abea66 2023-03-22 17:53:57,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ac91e00f0c5749d8ba63eb2eb9abea66, entries=150, sequenceid=420, filesize=11.9 K 2023-03-22 17:53:57,915 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~451.27 KB/462096, currentSize=26.84 KB/27480 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 877ms, sequenceid=420, compaction requested=true 2023-03-22 17:53:57,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:57,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:57,915 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:53:57,915 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:53:57,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:57,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:57,917 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64018 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:53:57,917 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:53:57,917 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:53:57,917 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,917 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:53:57,917 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,917 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5830f3124e0444d48887c06aa2723048, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/dea26a033b99477db4dbac7b72c28d3f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/54f6552cb91d444080768d0940a1f1a7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a581c2356e6a410e9be965a1d9a7fd29, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/943be75650f848bea8329a2ad27f85ed] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=62.5 K 2023-03-22 17:53:57,918 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/429b55bde83a4741864c75e7cbc6126b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/9abcfc68cb2e4dc1984e9562ec737910, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f1171554ade644d48d384e9cfd3143ef, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e17b86b27c394df8b12c2054936c09db, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/93605b870d0044398df72fe9702fe8dd] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=60.1 K 2023-03-22 17:53:57,918 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5830f3124e0444d48887c06aa2723048, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679507633213 2023-03-22 17:53:57,918 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 429b55bde83a4741864c75e7cbc6126b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679507633213 2023-03-22 17:53:57,918 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting dea26a033b99477db4dbac7b72c28d3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679507633858 2023-03-22 17:53:57,919 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9abcfc68cb2e4dc1984e9562ec737910, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679507633858 2023-03-22 17:53:57,919 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 54f6552cb91d444080768d0940a1f1a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679507634074 2023-03-22 17:53:57,919 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f1171554ade644d48d384e9cfd3143ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679507634074 2023-03-22 17:53:57,920 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a581c2356e6a410e9be965a1d9a7fd29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507635216 2023-03-22 17:53:57,920 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e17b86b27c394df8b12c2054936c09db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507635216 2023-03-22 17:53:57,920 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 943be75650f848bea8329a2ad27f85ed, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1679507636409 2023-03-22 17:53:57,920 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 93605b870d0044398df72fe9702fe8dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1679507636409 2023-03-22 17:53:57,935 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#72 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:57,939 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#73 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:57,971 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:57,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:57,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:57,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:57,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:57,974 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:57,974 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,974 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:57,974 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:57,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:57,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:57,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:57,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:57,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:57,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:57,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:57,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:57,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:57,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:57,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:57,978 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:57,978 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:57,979 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-22 17:53:57,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:57,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:57,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:57,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:57,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:57,979 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:57,992 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/18795c57df0b4591b7144f3f60840401 2023-03-22 17:53:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:58,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:58,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507638175, max timestamp=1679507638186Num uniques 859; 2023-03-22 17:53:58,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:53:58,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507638175, max timestamp=1679507638186Num uniques 859; 2023-03-22 17:53:58,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:53:58,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507638175, max timestamp=1679507638186Num uniques 859; 2023-03-22 17:53:58,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:53:58,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507638205, max timestamp=1679507638222Num uniques 1068; 2023-03-22 17:53:58,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507638205, max timestamp=1679507638222Num uniques 1068; 2023-03-22 17:53:58,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-22 17:53:58,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-22 17:53:58,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507638205, max timestamp=1679507638222Num uniques 1068; 2023-03-22 17:53:58,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-22 17:53:58,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507698255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679507638237, max timestamp=1679507638261Num uniques 1127; 2023-03-22 17:53:58,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-22 17:53:58,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679507638237, max timestamp=1679507638261Num uniques 1127; 2023-03-22 17:53:58,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-22 17:53:58,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679507638237, max timestamp=1679507638261Num uniques 1127; 2023-03-22 17:53:58,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-22 17:53:58,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679507638285, max timestamp=1679507638309Num uniques 1186; 2023-03-22 17:53:58,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-03-22 17:53:58,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679507638285, max timestamp=1679507638309Num uniques 1186; 2023-03-22 17:53:58,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-03-22 17:53:58,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679507638285, max timestamp=1679507638309Num uniques 1186; 2023-03-22 17:53:58,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-03-22 17:53:58,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507638341, max timestamp=1679507638348Num uniques 1145; 2023-03-22 17:53:58,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-22 17:53:58,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507638341, max timestamp=1679507638348Num uniques 1145; 2023-03-22 17:53:58,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-22 17:53:58,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507638341, max timestamp=1679507638348Num uniques 1145; 2023-03-22 17:53:58,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-22 17:53:58,361 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/24bd830bd7504a18b89ba162cba57850 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/24bd830bd7504a18b89ba162cba57850 2023-03-22 17:53:58,361 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/478f17600c7540e58d902f0129ce9381 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/478f17600c7540e58d902f0129ce9381 2023-03-22 17:53:58,372 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 478f17600c7540e58d902f0129ce9381(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:58,372 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:58,372 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=11, startTime=1679507637915; duration=0sec 2023-03-22 17:53:58,372 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:58,372 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:53:58,373 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 24bd830bd7504a18b89ba162cba57850(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:58,373 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:58,373 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=11, startTime=1679507637915; duration=0sec 2023-03-22 17:53:58,373 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:58,375 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:53:58,375 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:53:58,375 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:58,376 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3bcbfbea38e0489aa19505f818647347, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a60076075f59461ab05036dabb15377b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/feedb777e5934374be208d698af281d6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/0f6f6103adb54c85a74bffb5a5938065, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ac91e00f0c5749d8ba63eb2eb9abea66] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=60.1 K 2023-03-22 17:53:58,377 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3bcbfbea38e0489aa19505f818647347, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679507633213 2023-03-22 17:53:58,378 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a60076075f59461ab05036dabb15377b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679507633858 2023-03-22 17:53:58,378 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting feedb777e5934374be208d698af281d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679507634074 2023-03-22 17:53:58,379 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0f6f6103adb54c85a74bffb5a5938065, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507635216 2023-03-22 17:53:58,380 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ac91e00f0c5749d8ba63eb2eb9abea66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1679507636409 2023-03-22 17:53:58,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507698394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679507638378, max timestamp=1679507638393Num uniques 1104; 2023-03-22 17:53:58,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679507638378, max timestamp=1679507638393Num uniques 1104; 2023-03-22 17:53:58,397 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:58,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-22 17:53:58,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679507638378, max timestamp=1679507638393Num uniques 1104; 2023-03-22 17:53:58,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-22 17:53:58,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-22 17:53:58,405 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/f577e021bfe44057bb7474ade816282b 2023-03-22 17:53:58,424 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/69abe2351c5f465c964610067cf85bdf as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/69abe2351c5f465c964610067cf85bdf 2023-03-22 17:53:58,435 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 69abe2351c5f465c964610067cf85bdf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:58,435 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:58,435 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=11, startTime=1679507637915; duration=0sec 2023-03-22 17:53:58,435 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:58,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507698469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679507638440, max timestamp=1679507638469Num uniques 1113; 2023-03-22 17:53:58,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679507638440, max timestamp=1679507638469Num uniques 1113; 2023-03-22 17:53:58,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-22 17:53:58,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679507638440, max timestamp=1679507638469Num uniques 1113; 2023-03-22 17:53:58,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-22 17:53:58,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-22 17:53:58,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679507638495, max timestamp=1679507638509Num uniques 1072; 2023-03-22 17:53:58,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-22 17:53:58,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679507638495, max timestamp=1679507638509Num uniques 1072; 2023-03-22 17:53:58,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-22 17:53:58,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679507638495, max timestamp=1679507638509Num uniques 1072; 2023-03-22 17:53:58,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-22 17:53:58,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679507638526, max timestamp=1679507638533Num uniques 1081; 2023-03-22 17:53:58,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679507638526, max timestamp=1679507638533Num uniques 1081; 2023-03-22 17:53:58,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-22 17:53:58,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-22 17:53:58,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679507638526, max timestamp=1679507638533Num uniques 1081; 2023-03-22 17:53:58,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-22 17:53:58,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679507638550, max timestamp=1679507638560Num uniques 1040; 2023-03-22 17:53:58,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-03-22 17:53:58,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679507638550, max timestamp=1679507638560Num uniques 1040; 2023-03-22 17:53:58,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-03-22 17:53:58,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679507638550, max timestamp=1679507638560Num uniques 1040; 2023-03-22 17:53:58,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-03-22 17:53:58,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507698600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=499, cellSize=22854, totalHeapSize=52136, min timestamp=1679507638571, max timestamp=1679507638602Num uniques 1149; 2023-03-22 17:53:58,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=22854, suffix heap size=147464,new segment heap size=52136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=499 2023-03-22 17:53:58,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=499, cellSize=22854, totalHeapSize=52136, min timestamp=1679507638571, max timestamp=1679507638602Num uniques 1149; 2023-03-22 17:53:58,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=22854, suffix heap size=147464,new segment heap size=52136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=499 2023-03-22 17:53:58,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=499, cellSize=22854, totalHeapSize=52136, min timestamp=1679507638571, max timestamp=1679507638602Num uniques 1149; 2023-03-22 17:53:58,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=22854, suffix heap size=147464,new segment heap size=52136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=499 2023-03-22 17:53:58,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507698623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507698623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507698624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507698705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507698725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507698728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507698728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45284 deadline: 1679507698738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,741 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:53:58,854 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/e77179c2b24a43ac8d3edaff1e8a7540 2023-03-22 17:53:58,860 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/18795c57df0b4591b7144f3f60840401 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/18795c57df0b4591b7144f3f60840401 2023-03-22 17:53:58,865 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/18795c57df0b4591b7144f3f60840401, entries=150, sequenceid=427, filesize=11.9 K 2023-03-22 17:53:58,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/f577e021bfe44057bb7474ade816282b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f577e021bfe44057bb7474ade816282b 2023-03-22 17:53:58,872 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f577e021bfe44057bb7474ade816282b, entries=150, sequenceid=427, filesize=11.9 K 2023-03-22 17:53:58,873 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/e77179c2b24a43ac8d3edaff1e8a7540 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/e77179c2b24a43ac8d3edaff1e8a7540 2023-03-22 17:53:58,878 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/e77179c2b24a43ac8d3edaff1e8a7540, entries=150, sequenceid=427, filesize=11.9 K 2023-03-22 17:53:58,879 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 900ms, sequenceid=427, compaction requested=false 2023-03-22 17:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:58,879 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:58,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:53:58,911 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=167.72 KB heapSize=452.72 KB 2023-03-22 17:53:58,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:58,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:58,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:58,911 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:58,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:58,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:58,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:53:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:58,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:58,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:58,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15e2b4c4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:58,914 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:58,915 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:58,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507698917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/438880eed8b34806b07ef79304ca115e 2023-03-22 17:53:58,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507698929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507698931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507698932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:58,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:58,950 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:58,951 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:59,012 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:59,012 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:59,012 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:59,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:59,019 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:59,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:59,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:59,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:59,019 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:59,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507699021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,073 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,073 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,073 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:59,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,075 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:59,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:59,127 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:59,128 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:59,128 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:59,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:59,128 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:59,128 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:59,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:59,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:59,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507699224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,228 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:59,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:59,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:59,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:59,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:59,232 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:59,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:59,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:59,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:59,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507699233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507699235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507699235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,289 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:59,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:59,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:59,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:59,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:59,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:59,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:59,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:59,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:59,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:59,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:59,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:59,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,335 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@181f4ae5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:59,337 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:59,337 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,340 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/2c45b2f3890244bdada917465a0935e4 2023-03-22 17:53:59,358 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:59,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:59,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:59,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:59,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:59,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:59,451 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:59,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:59,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:59,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:59,451 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:59,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:59,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:59,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:53:59,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:59,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:59,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:59,476 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:59,476 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:59,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:59,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507699529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,576 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:59,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:59,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:59,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:59,582 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:59,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:59,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:59,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:59,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,585 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:59,585 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:59,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:59,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:59,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:59,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:59,587 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:59,587 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:59,588 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,588 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:59,588 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:59,588 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:59,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:59,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:59,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14901e1e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:59,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:59,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:59,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:59,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:59,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:59,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:59,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:59,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:59,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:59,600 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:59,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:59,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:59,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:59,601 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:59,604 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,604 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,604 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:59,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:59,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:59,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:59,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:59,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:59,622 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:59,622 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:59,722 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:59,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:59,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:59,737 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:59,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:59,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:59,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:59,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507699739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:59,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:59,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:59,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507699739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:53:59,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507699745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:59,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:59,757 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,757 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:59,757 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:53:59,757 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:53:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:53:59,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f6724cb[Count = 0] remaining members to acquire global barrier 2023-03-22 17:53:59,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:53:59,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:53:59,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:53:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:53:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:53:59,770 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:53:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:53:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/7f5871e21069424a909b71c797f51eca 2023-03-22 17:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:53:59,775 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:53:59,775 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:53:59,775 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:53:59,775 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:53:59,775 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:53:59,780 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,780 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,780 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:53:59,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:53:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/438880eed8b34806b07ef79304ca115e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/438880eed8b34806b07ef79304ca115e 2023-03-22 17:53:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:53:59,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:53:59,786 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:53:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/438880eed8b34806b07ef79304ca115e, entries=200, sequenceid=637, filesize=14.3 K 2023-03-22 17:53:59,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:59,790 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:53:59,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:53:59,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:53:59,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:53:59,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:59,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,790 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:53:59,791 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:53:59,791 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,791 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,791 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:53:59,791 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:53:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:53:59,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/2c45b2f3890244bdada917465a0935e4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/2c45b2f3890244bdada917465a0935e4 2023-03-22 17:53:59,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/2c45b2f3890244bdada917465a0935e4, entries=150, sequenceid=637, filesize=11.9 K 2023-03-22 17:53:59,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/7f5871e21069424a909b71c797f51eca as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/7f5871e21069424a909b71c797f51eca 2023-03-22 17:53:59,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/7f5871e21069424a909b71c797f51eca, entries=150, sequenceid=637, filesize=11.9 K 2023-03-22 17:53:59,804 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~452.67 KB/463536, currentSize=20.13 KB/20610 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 893ms, sequenceid=637, compaction requested=true 2023-03-22 17:53:59,804 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:59,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:59,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:59,804 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:59,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:59,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:59,805 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39816 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:59,805 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:53:59,805 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:59,806 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,806 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:53:59,806 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/24bd830bd7504a18b89ba162cba57850, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/18795c57df0b4591b7144f3f60840401, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/438880eed8b34806b07ef79304ca115e] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=38.9 K 2023-03-22 17:53:59,806 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,806 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/478f17600c7540e58d902f0129ce9381, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f577e021bfe44057bb7474ade816282b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/2c45b2f3890244bdada917465a0935e4] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=36.5 K 2023-03-22 17:53:59,806 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 24bd830bd7504a18b89ba162cba57850, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1679507636409 2023-03-22 17:53:59,806 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 478f17600c7540e58d902f0129ce9381, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1679507636409 2023-03-22 17:53:59,807 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 18795c57df0b4591b7144f3f60840401, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679507637039 2023-03-22 17:53:59,807 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f577e021bfe44057bb7474ade816282b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679507637039 2023-03-22 17:53:59,808 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 438880eed8b34806b07ef79304ca115e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679507638613 2023-03-22 17:53:59,808 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2c45b2f3890244bdada917465a0935e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679507638613 2023-03-22 17:53:59,818 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:59,825 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:59,843 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/f19951dea1de4696851c33781da64eff as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f19951dea1de4696851c33781da64eff 2023-03-22 17:53:59,849 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into f19951dea1de4696851c33781da64eff(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:59,849 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:59,849 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=13, startTime=1679507639804; duration=0sec 2023-03-22 17:53:59,849 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:53:59,849 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:53:59,850 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:53:59,851 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:53:59,851 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,851 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/69abe2351c5f465c964610067cf85bdf, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/e77179c2b24a43ac8d3edaff1e8a7540, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/7f5871e21069424a909b71c797f51eca] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=36.5 K 2023-03-22 17:53:59,851 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 69abe2351c5f465c964610067cf85bdf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1679507636409 2023-03-22 17:53:59,852 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e77179c2b24a43ac8d3edaff1e8a7540, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1679507637039 2023-03-22 17:53:59,852 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 7f5871e21069424a909b71c797f51eca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679507638613 2023-03-22 17:53:59,860 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#83 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:53:59,873 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/ce59b2b16bb64e0f9162a4ec862408fa as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ce59b2b16bb64e0f9162a4ec862408fa 2023-03-22 17:53:59,881 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into ce59b2b16bb64e0f9162a4ec862408fa(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:53:59,881 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:53:59,881 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=13, startTime=1679507639804; duration=0sec 2023-03-22 17:53:59,881 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:53:59,891 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:53:59,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:53:59,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:53:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:53:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:53:59,894 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:53:59,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:59,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:53:59,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:53:59,895 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,895 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:53:59,896 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:53:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:53:59,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:53:59,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:53:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:53:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:53:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:53:59,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:53:59,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:53:59,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:53:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:53:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:53:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:53:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:53:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:53:59,898 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:53:59,898 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:53:59,898 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-22 17:53:59,898 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:53:59,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:59,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:53:59,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:53:59,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:53:59,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:00,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:54:00,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:00,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507640057, max timestamp=1679507640073Num uniques 859; 2023-03-22 17:54:00,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:00,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507640057, max timestamp=1679507640073Num uniques 859; 2023-03-22 17:54:00,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507640057, max timestamp=1679507640073Num uniques 859; 2023-03-22 17:54:00,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:00,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:00,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507640105, max timestamp=1679507640117Num uniques 1068; 2023-03-22 17:54:00,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-22 17:54:00,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507640105, max timestamp=1679507640117Num uniques 1068; 2023-03-22 17:54:00,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-22 17:54:00,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507640105, max timestamp=1679507640117Num uniques 1068; 2023-03-22 17:54:00,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-22 17:54:00,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679507640139, max timestamp=1679507640171Num uniques 1077; 2023-03-22 17:54:00,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-22 17:54:00,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679507640139, max timestamp=1679507640171Num uniques 1077; 2023-03-22 17:54:00,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-22 17:54:00,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679507640139, max timestamp=1679507640171Num uniques 1077; 2023-03-22 17:54:00,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-22 17:54:00,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507640206, max timestamp=1679507640235Num uniques 1086; 2023-03-22 17:54:00,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507640206, max timestamp=1679507640235Num uniques 1086; 2023-03-22 17:54:00,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-22 17:54:00,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-22 17:54:00,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507640206, max timestamp=1679507640235Num uniques 1086; 2023-03-22 17:54:00,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-22 17:54:00,246 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/1faeaaafb64641d9acf4b49533b90b19 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1faeaaafb64641d9acf4b49533b90b19 2023-03-22 17:54:00,265 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 1faeaaafb64641d9acf4b49533b90b19(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:00,265 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:00,265 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=13, startTime=1679507639804; duration=0sec 2023-03-22 17:54:00,265 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:00,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507640260, max timestamp=1679507640293Num uniques 1095; 2023-03-22 17:54:00,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-22 17:54:00,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507640260, max timestamp=1679507640293Num uniques 1095; 2023-03-22 17:54:00,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507640260, max timestamp=1679507640293Num uniques 1095; 2023-03-22 17:54:00,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-22 17:54:00,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-22 17:54:00,308 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/7c4587a07c8640e19eadd63d0a9c8ddc 2023-03-22 17:54:00,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679507640366, max timestamp=1679507640394Num uniques 1104; 2023-03-22 17:54:00,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679507640366, max timestamp=1679507640394Num uniques 1104; 2023-03-22 17:54:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679507640366, max timestamp=1679507640394Num uniques 1104; 2023-03-22 17:54:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-22 17:54:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-22 17:54:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-22 17:54:00,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679507640444, max timestamp=1679507640461Num uniques 1013; 2023-03-22 17:54:00,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679507640444, max timestamp=1679507640461Num uniques 1013; 2023-03-22 17:54:00,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=6870, suffix heap size=133320,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=150 2023-03-22 17:54:00,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-22 17:54:00,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679507640444, max timestamp=1679507640461Num uniques 1013; 2023-03-22 17:54:00,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-22 17:54:00,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679507640480, max timestamp=1679507640512Num uniques 1022; 2023-03-22 17:54:00,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-22 17:54:00,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679507640484, max timestamp=1679507640512Num uniques 1022; 2023-03-22 17:54:00,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-22 17:54:00,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679507640484, max timestamp=1679507640512Num uniques 1009; 2023-03-22 17:54:00,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7872, suffix heap size=132904,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=172 2023-03-22 17:54:00,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679507640548, max timestamp=1679507640557Num uniques 1031; 2023-03-22 17:54:00,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-22 17:54:00,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679507640548, max timestamp=1679507640557Num uniques 1081; 2023-03-22 17:54:00,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679507640548, max timestamp=1679507640557Num uniques 1031; 2023-03-22 17:54:00,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-22 17:54:00,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-22 17:54:00,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679507640594, max timestamp=1679507640610Num uniques 1090; 2023-03-22 17:54:00,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-22 17:54:00,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679507640594, max timestamp=1679507640610Num uniques 1090; 2023-03-22 17:54:00,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-22 17:54:00,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679507640594, max timestamp=1679507640610Num uniques 1090; 2023-03-22 17:54:00,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-22 17:54:00,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679507640651, max timestamp=1679507640684Num uniques 1099; 2023-03-22 17:54:00,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-03-22 17:54:00,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679507640651, max timestamp=1679507640684Num uniques 1099; 2023-03-22 17:54:00,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-03-22 17:54:00,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679507640651, max timestamp=1679507640684Num uniques 1099; 2023-03-22 17:54:00,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-03-22 17:54:00,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679507640711, max timestamp=1679507640747Num uniques 1158; 2023-03-22 17:54:00,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679507640711, max timestamp=1679507640747Num uniques 1158; 2023-03-22 17:54:00,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-03-22 17:54:00,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679507640711, max timestamp=1679507640747Num uniques 1158; 2023-03-22 17:54:00,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-03-22 17:54:00,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-03-22 17:54:00,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1679507640758, max timestamp=1679507640769Num uniques 1067; 2023-03-22 17:54:00,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=16802, suffix heap size=138936,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=367 2023-03-22 17:54:00,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1679507640758, max timestamp=1679507640769Num uniques 1067; 2023-03-22 17:54:00,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1679507640758, max timestamp=1679507640769Num uniques 1067; 2023-03-22 17:54:00,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=16802, suffix heap size=138936,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=367 2023-03-22 17:54:00,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=16802, suffix heap size=138936,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=367 2023-03-22 17:54:00,773 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/ed3b1ce1af3f4b36ae25f60500228c0a 2023-03-22 17:54:00,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507700804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:00,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507700807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679507640790, max timestamp=1679507640806Num uniques 1226; 2023-03-22 17:54:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=8056, suffix heap size=155472,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=176 2023-03-22 17:54:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679507640790, max timestamp=1679507640806Num uniques 1226; 2023-03-22 17:54:00,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=8056, suffix heap size=155472,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=176 2023-03-22 17:54:00,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679507640790, max timestamp=1679507640806Num uniques 1226; 2023-03-22 17:54:00,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=8056, suffix heap size=155472,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=176 2023-03-22 17:54:00,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=385, cellSize=17630, totalHeapSize=40280, min timestamp=1679507640846, max timestamp=1679507640861Num uniques 1035; 2023-03-22 17:54:00,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=17630, suffix heap size=135608,new segment heap size=40280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=385 2023-03-22 17:54:00,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679507640852, max timestamp=1679507640861Num uniques 1035; 2023-03-22 17:54:00,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-22 17:54:00,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679507640852, max timestamp=1679507640861Num uniques 1035; 2023-03-22 17:54:00,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-22 17:54:00,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679507640886, max timestamp=1679507640894Num uniques 1244; 2023-03-22 17:54:00,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56974, new segment data size=8884, suffix heap size=157344,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1244, new segment cells count=194 2023-03-22 17:54:00,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679507640886, max timestamp=1679507640894Num uniques 1044; 2023-03-22 17:54:00,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679507640886, max timestamp=1679507640894Num uniques 1044; 2023-03-22 17:54:00,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-22 17:54:00,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-22 17:54:00,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1679507640911, max timestamp=1679507640924Num uniques 1053; 2023-03-22 17:54:00,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=13875, suffix heap size=137480,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=303 2023-03-22 17:54:00,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679507640919, max timestamp=1679507640924Num uniques 1053; 2023-03-22 17:54:00,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-22 17:54:00,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679507640919, max timestamp=1679507640925Num uniques 1053; 2023-03-22 17:54:00,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-22 17:54:00,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1679507640939, max timestamp=1679507640947Num uniques 1012; 2023-03-22 17:54:00,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14280, totalHeapSize=32688, min timestamp=1679507640939, max timestamp=1679507640947Num uniques 1012; 2023-03-22 17:54:00,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1679507640939, max timestamp=1679507640947Num uniques 1162; 2023-03-22 17:54:00,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=14282, suffix heap size=133216,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=312 2023-03-22 17:54:00,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=14282, suffix heap size=148816,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=312 2023-03-22 17:54:00,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46340, new segment data size=14280, suffix heap size=133216,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=312 2023-03-22 17:54:00,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679507640957, max timestamp=1679507640968Num uniques 1171; 2023-03-22 17:54:00,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=12406, suffix heap size=149752,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=271 2023-03-22 17:54:00,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679507640957, max timestamp=1679507640968Num uniques 1171; 2023-03-22 17:54:00,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=12406, suffix heap size=149752,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=271 2023-03-22 17:54:00,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679507640957, max timestamp=1679507640968Num uniques 1171; 2023-03-22 17:54:00,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=12406, suffix heap size=149752,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=271 2023-03-22 17:54:00,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:00,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507700983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:00,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:00,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507700986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:00,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=430, cellSize=19690, totalHeapSize=44960, min timestamp=1679507640976, max timestamp=1679507640985Num uniques 1130; 2023-03-22 17:54:00,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=19690, suffix heap size=145488,new segment heap size=44960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=430 2023-03-22 17:54:00,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=430, cellSize=19690, totalHeapSize=44960, min timestamp=1679507640976, max timestamp=1679507640985Num uniques 1130; 2023-03-22 17:54:00,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=19690, suffix heap size=145488,new segment heap size=44960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=430 2023-03-22 17:54:00,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=430, cellSize=19689, totalHeapSize=44960, min timestamp=1679507640976, max timestamp=1679507640986Num uniques 1130; 2023-03-22 17:54:00,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51749, new segment data size=19689, suffix heap size=145488,new segment heap size=44960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=430 2023-03-22 17:54:01,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507701008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507701009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507701088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507701089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507701111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507701111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,196 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/02e88216bd8a4bf3ae686da8ed744dba 2023-03-22 17:54:01,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/7c4587a07c8640e19eadd63d0a9c8ddc as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7c4587a07c8640e19eadd63d0a9c8ddc 2023-03-22 17:54:01,208 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7c4587a07c8640e19eadd63d0a9c8ddc, entries=100, sequenceid=645, filesize=9.5 K 2023-03-22 17:54:01,209 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/ed3b1ce1af3f4b36ae25f60500228c0a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/ed3b1ce1af3f4b36ae25f60500228c0a 2023-03-22 17:54:01,215 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/ed3b1ce1af3f4b36ae25f60500228c0a, entries=100, sequenceid=645, filesize=9.5 K 2023-03-22 17:54:01,216 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/02e88216bd8a4bf3ae686da8ed744dba as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/02e88216bd8a4bf3ae686da8ed744dba 2023-03-22 17:54:01,220 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/02e88216bd8a4bf3ae686da8ed744dba, entries=100, sequenceid=645, filesize=9.5 K 2023-03-22 17:54:01,221 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=167.72 KB/171750 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 1323ms, sequenceid=645, compaction requested=false 2023-03-22 17:54:01,221 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:01,221 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:01,225 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f23fd77[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:01,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:01,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,233 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,234 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:01,236 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:01,239 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:01,239 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:01,239 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:01,239 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:01,239 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:01,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:01,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,249 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,249 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:01,250 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:01,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:54:01,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=174.43 KB heapSize=479.81 KB 2023-03-22 17:54:01,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:54:01,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:54:01,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:54:01,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:54:01,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:54:01,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:54:01,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507701296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507701297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507701314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507701315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,350 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:01,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:01,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:01,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:01,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:01,355 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:01,355 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,355 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:01,355 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:01,355 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,355 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,358 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:01,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:01,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:01,361 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,361 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:01,361 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:01,361 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:01,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:01,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:01,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:01,363 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,363 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,364 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,364 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:01,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d09de14[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:01,365 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:01,365 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:01,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:01,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:01,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:01,371 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:01,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:01,375 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:01,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:01,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:01,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:01,375 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:01,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,380 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,382 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:01,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:01,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:01,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,385 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:01,385 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:01,385 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,385 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507701401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507701401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,485 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:01,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:01,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:01,489 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:01,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:01,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:01,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:01,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:01,493 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:01,494 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,494 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:01,494 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:01,494 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:01,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397d5545[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:01,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:01,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:01,500 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:01,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:01,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:01,502 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:01,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:01,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:01,505 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:01,505 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:01,505 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:01,505 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:01,505 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:01,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,511 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,513 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:01,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,515 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:01,516 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,516 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:01,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507701603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507701604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,616 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:01,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:01,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:01,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:01,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:01,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507701619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,620 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:01,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:01,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:01,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507701620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,623 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,623 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:01,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:01,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:01,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,627 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:01,631 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:01,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4a6fdd[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:01,631 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:01,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:01,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:01,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:01,636 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:01,636 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:01,636 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:01,636 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:01,636 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:01,638 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,638 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,638 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,638 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,640 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:01,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:01,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:01,652 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:01,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=1010 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/1eb0de01b71e4d839bb7edafb65f5539 2023-03-22 17:54:01,753 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:01,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:01,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:01,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:01,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:01,756 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:01,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:01,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:01,757 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,757 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,758 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:01,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:01,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:01,760 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,760 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:01,760 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:01,760 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:01,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:01,762 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:01,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:01,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd2eb06[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:01,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,765 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:01,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:01,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:01,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:01,770 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:01,770 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:01,770 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:01,770 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:01,770 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:01,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,773 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,774 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,776 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,776 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,776 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:01,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:01,776 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,776 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,776 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:01,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:01,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:01,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:01,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:01,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:01,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:01,880 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:01,880 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,880 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:01,880 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:01,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,882 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:01,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:01,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:01,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:01,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:01,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:01,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c486f96[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:01,888 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:01,889 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,890 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,890 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:01,890 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:01,890 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:01,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:01,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:01,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:01,894 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:01,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:01,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:01,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:01,894 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:01,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:01,896 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:01,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,896 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,898 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,901 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,901 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,901 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:01,901 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:01,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:01,901 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:01,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:01,901 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:01,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:01,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:01,901 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:01,902 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:01,902 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:01,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507701906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:01,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:01,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507701907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,002 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:02,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:02,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:02,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:02,006 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:02,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:02,010 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,010 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:02,010 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:02,010 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:02,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:02,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f4cca27[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:02,015 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:02,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:02,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:02,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:02,017 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,017 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:02,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:02,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:02,027 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:02,027 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:02,027 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:02,027 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:02,027 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:02,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,029 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,036 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,037 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:02,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:02,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:02,038 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:02,038 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:02,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=1010 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/716885e7816545da8b124e48eeea20fd 2023-03-22 17:54:02,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:02,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507702122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:02,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507702124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,138 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:02,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:02,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:02,146 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:02,146 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,146 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:02,146 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:02,147 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,147 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,152 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,152 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,153 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,153 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:02,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:02,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:02,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:02,157 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,157 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:02,157 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:02,157 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:02,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,159 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:02,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33646c94[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:02,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:02,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,164 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:02,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:02,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:02,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:02,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:02,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:02,173 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:02,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:02,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:02,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:02,173 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:02,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,177 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:02,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:02,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,184 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:02,184 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:02,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:02,284 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:02,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:02,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:02,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:02,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:02,288 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:02,288 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,288 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:02,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:02,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,290 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:02,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:02,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:02,292 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,292 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:02,293 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:02,293 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:02,296 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:02,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b662b25[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:02,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:02,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,299 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:02,299 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,299 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:02,300 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:02,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,301 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:02,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:02,314 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:02,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:02,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:02,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:02,314 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:02,316 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,316 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,320 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:02,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,324 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:02,324 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:02,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507702414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:02,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507702414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,424 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:02,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:02,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:02,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:02,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:02,427 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:02,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:02,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:02,428 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,428 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,429 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,429 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,430 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,430 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:02,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:02,431 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,431 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:02,431 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:02,431 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:02,433 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,433 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,434 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,434 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46dc700[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:02,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:02,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,437 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:02,437 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,437 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:02,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,439 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,439 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,439 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:02,441 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:02,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:02,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:02,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:02,441 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:02,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:02,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:02,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,447 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,448 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:02,448 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,448 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=1010 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/8fa6b7e494084270bc496c4a95277548 2023-03-22 17:54:02,548 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:02,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:02,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:02,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:02,551 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:02,551 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,551 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:02,551 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:02,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,552 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/1eb0de01b71e4d839bb7edafb65f5539 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1eb0de01b71e4d839bb7edafb65f5539 2023-03-22 17:54:02,553 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,553 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,554 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,554 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:02,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:02,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:02,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:02,555 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,555 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:02,556 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:02,556 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:02,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1eb0de01b71e4d839bb7edafb65f5539, entries=150, sequenceid=1010, filesize=11.9 K 2023-03-22 17:54:02,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/716885e7816545da8b124e48eeea20fd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/716885e7816545da8b124e48eeea20fd 2023-03-22 17:54:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,559 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db92858[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:02,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:02,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,573 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:02,573 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,573 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:02,574 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,575 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:02,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:02,578 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:02,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:02,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:02,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:02,578 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:02,580 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,580 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/716885e7816545da8b124e48eeea20fd, entries=150, sequenceid=1010, filesize=11.9 K 2023-03-22 17:54:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,580 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/8fa6b7e494084270bc496c4a95277548 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8fa6b7e494084270bc496c4a95277548 2023-03-22 17:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,584 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:02,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:02,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,588 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:02,588 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:02,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8fa6b7e494084270bc496c4a95277548, entries=150, sequenceid=1010, filesize=11.9 K 2023-03-22 17:54:02,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:02,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~479.77 KB/491280, currentSize=13.42 KB/13740 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 1296ms, sequenceid=1010, compaction requested=true 2023-03-22 17:54:02,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:02,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:02,589 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:02,589 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:02,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:02,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:02,592 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:02,592 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:54:02,592 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,592 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:02,592 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f19951dea1de4696851c33781da64eff, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7c4587a07c8640e19eadd63d0a9c8ddc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1eb0de01b71e4d839bb7edafb65f5539] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=34.2 K 2023-03-22 17:54:02,593 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:54:02,593 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,593 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1faeaaafb64641d9acf4b49533b90b19, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/ed3b1ce1af3f4b36ae25f60500228c0a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/716885e7816545da8b124e48eeea20fd] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=34.2 K 2023-03-22 17:54:02,594 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f19951dea1de4696851c33781da64eff, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679507638613 2023-03-22 17:54:02,595 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1faeaaafb64641d9acf4b49533b90b19, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679507638613 2023-03-22 17:54:02,595 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 7c4587a07c8640e19eadd63d0a9c8ddc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1679507638914 2023-03-22 17:54:02,596 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ed3b1ce1af3f4b36ae25f60500228c0a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1679507638914 2023-03-22 17:54:02,596 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1eb0de01b71e4d839bb7edafb65f5539, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1679507641007 2023-03-22 17:54:02,596 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 716885e7816545da8b124e48eeea20fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1679507641007 2023-03-22 17:54:02,610 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:02,612 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:02,655 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/c93fea7fc5fc40849da909dfbe9e74bd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/c93fea7fc5fc40849da909dfbe9e74bd 2023-03-22 17:54:02,663 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into c93fea7fc5fc40849da909dfbe9e74bd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:02,663 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:02,663 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=13, startTime=1679507642589; duration=0sec 2023-03-22 17:54:02,664 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:02,664 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:02,666 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:02,666 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:54:02,666 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,666 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ce59b2b16bb64e0f9162a4ec862408fa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/02e88216bd8a4bf3ae686da8ed744dba, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8fa6b7e494084270bc496c4a95277548] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=34.2 K 2023-03-22 17:54:02,666 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ce59b2b16bb64e0f9162a4ec862408fa, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679507638613 2023-03-22 17:54:02,668 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 02e88216bd8a4bf3ae686da8ed744dba, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1679507638914 2023-03-22 17:54:02,672 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 8fa6b7e494084270bc496c4a95277548, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1679507641007 2023-03-22 17:54:02,680 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:02,688 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:02,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:02,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:02,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:02,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:02,692 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:02,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:02,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:02,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,695 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:02,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:02,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:02,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:02,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:02,698 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:02,698 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:02,698 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,698 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:02,698 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-22 17:54:02,699 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:54:02,699 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:02,699 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:54:02,699 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:02,699 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:54:02,699 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:02,707 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1016 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/a5b1c0609817428b83a7750a93c8b22f 2023-03-22 17:54:02,721 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1016 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/a22374b639424bfdb7f80db3f1121660 2023-03-22 17:54:02,741 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1016 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/f235b982cf6e4d49a4d5382f5288a834 2023-03-22 17:54:02,746 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/a5b1c0609817428b83a7750a93c8b22f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a5b1c0609817428b83a7750a93c8b22f 2023-03-22 17:54:02,751 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a5b1c0609817428b83a7750a93c8b22f, entries=50, sequenceid=1016, filesize=7.1 K 2023-03-22 17:54:02,752 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/a22374b639424bfdb7f80db3f1121660 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a22374b639424bfdb7f80db3f1121660 2023-03-22 17:54:02,756 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a22374b639424bfdb7f80db3f1121660, entries=50, sequenceid=1016, filesize=7.1 K 2023-03-22 17:54:02,757 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/f235b982cf6e4d49a4d5382f5288a834 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/f235b982cf6e4d49a4d5382f5288a834 2023-03-22 17:54:02,761 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/f235b982cf6e4d49a4d5382f5288a834, entries=50, sequenceid=1016, filesize=7.1 K 2023-03-22 17:54:02,762 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 64ms, sequenceid=1016, compaction requested=false 2023-03-22 17:54:02,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:02,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:02,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:02,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:02,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:02,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,764 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,765 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,765 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d29fdb7[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:02,767 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:02,767 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:02,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:02,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,771 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:02,771 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,771 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:02,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:02,774 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:02,774 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:02,774 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:02,774 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:02,774 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:02,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,784 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,784 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,786 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:02,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:02,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,790 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:02,790 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:02,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,890 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:02,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:02,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:02,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:02,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:02,894 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:02,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:02,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:02,895 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,895 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,897 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:02,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:02,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:02,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:02,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:02,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:02,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:02,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:02,899 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:02,901 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,902 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,902 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:02,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@571cca34[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:02,903 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:02,903 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,905 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,905 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:02,905 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:02,905 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:02,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:02,906 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:02,906 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,906 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:02,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:02,909 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:02,909 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:02,909 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:02,909 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:02,909 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:02,911 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,911 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,911 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,913 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,916 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:02,916 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:02,916 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:02,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:02,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:02,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,917 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:02,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:02,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:02,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,917 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:02,917 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:02,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:02,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:02,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,017 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:03,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:03,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:03,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:03,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:03,021 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:03,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:03,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:03,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,023 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,023 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,024 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:03,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:03,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:03,025 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,025 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:03,025 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:03,025 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:03,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d235210[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:03,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:03,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,034 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:03,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:03,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:03,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:03,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:03,039 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:03,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:03,042 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:03,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:03,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:03,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:03,043 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:03,045 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,045 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,045 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,046 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,047 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/0679bfb3d6d349469fb2b747c5bf4655 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/0679bfb3d6d349469fb2b747c5bf4655 2023-03-22 17:54:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,051 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,051 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,051 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:03,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:03,051 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,051 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,051 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:03,051 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:03,051 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:03,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,056 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 0679bfb3d6d349469fb2b747c5bf4655(size=12.9 K), total size for store is 20.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:03,056 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:03,056 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=13, startTime=1679507642589; duration=0sec 2023-03-22 17:54:03,056 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:03,098 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/aed902e6db2c4fefbcadfa0c0dded275 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/aed902e6db2c4fefbcadfa0c0dded275 2023-03-22 17:54:03,104 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into aed902e6db2c4fefbcadfa0c0dded275(size=12.9 K), total size for store is 20.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:03,104 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:03,104 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=13, startTime=1679507642589; duration=0sec 2023-03-22 17:54:03,105 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:03,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:54:03,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:54:03,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:54:03,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:03,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:54:03,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:03,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:54:03,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:03,152 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:03,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:03,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:03,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:03,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:03,154 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:03,155 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,155 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:03,155 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:03,155 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,155 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507643150, max timestamp=1679507643155Num uniques 859; 2023-03-22 17:54:03,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507703157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679507643150, max timestamp=1679507643155Num uniques 859; 2023-03-22 17:54:03,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-22 17:54:03,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:54:03,159 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,159 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507643150, max timestamp=1679507643155Num uniques 859; 2023-03-22 17:54:03,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:54:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,159 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,159 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:03,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:03,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:03,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:03,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:03,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:03,164 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:03,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39dc4604[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:03,166 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:03,166 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:03,169 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:03,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:03,172 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:03,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:03,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:03,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:03,172 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:03,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,176 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:03,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:03,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,180 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:03,180 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:03,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:03,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507703197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507703258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,280 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:03,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:03,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:03,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:03,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:03,284 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:03,284 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,284 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:03,284 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:03,284 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,284 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,286 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,286 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,286 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,286 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:03,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:03,288 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,288 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:03,288 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:03,288 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:03,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:03,290 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:03,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0ac0cd[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:03,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:03,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:03,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:03,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:03,293 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,293 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:03,295 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:03,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:03,298 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:03,298 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:03,298 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:03,298 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:03,298 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:03,300 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,300 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,300 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,300 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507703301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,306 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:03,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:03,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:03,307 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:03,307 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:03,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,407 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:03,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:03,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:03,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:03,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:03,410 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:03,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:03,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:03,411 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,411 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,412 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,412 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,413 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:03,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:03,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:03,414 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,414 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:03,414 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:03,414 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:03,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:03,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1af6e2f2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:03,419 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:03,419 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45350 deadline: 1679507703418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:03,420 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,420 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:03,421 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:03,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:03,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:03,424 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:03,424 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:03,424 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:03,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:03,425 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:03,426 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,426 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,426 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45348 deadline: 1679507703426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,429 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:03,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:03,436 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:03,436 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,436 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:03,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507703463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507703504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,536 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:03,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:03,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:03,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:03,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:03,540 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:03,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:03,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:03,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:03,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:03,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:03,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:03,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:03,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:03,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:03,544 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,544 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:03,544 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:03,544 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:03,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:03,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:03,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:03,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/1b78de980cda4106b4b573b3a4e22760 2023-03-22 17:54:03,545 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,545 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:03,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d62906e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:03,547 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:03,547 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:03,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:03,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:03,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:03,558 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:03,558 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,558 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:03,558 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:03,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:03,566 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:03,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:03,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:03,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:03,566 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:03,568 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,568 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,569 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,569 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,573 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:03,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,576 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:03,576 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:03,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:03,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/a0eb626f9b5f46e58fea6fc7777c01e8 2023-03-22 17:54:03,676 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:03,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:03,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:03,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:03,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:03,680 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:03,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:03,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:03,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,683 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,684 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:03,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:03,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:03,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:03,685 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,685 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:03,685 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:03,685 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:03,686 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:03,686 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,686 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:03,686 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:03,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,688 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,688 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:03,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:03,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66550da0[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:03,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:03,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:03,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:03,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:03,691 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,691 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:03,693 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:03,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,693 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,693 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:03,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:03,696 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:03,696 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:03,696 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:03,696 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:03,696 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:03,699 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,699 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,699 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,701 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:03,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,704 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,704 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,704 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,704 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:03,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45314 deadline: 1679507703766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,805 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:03,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:03,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:03,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:03,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:03,808 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:03,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:03,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:03,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,810 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:03,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45324 deadline: 1679507703809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:03,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:03,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:03,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:03,812 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,812 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. started... 2023-03-22 17:54:03,812 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. as already flushing 2023-03-22 17:54:03,812 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:03,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:03,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:03,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:03,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:03,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:03,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b24a50f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:03,816 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:03,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,817 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:55285 2023-03-22 17:54:03,817 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:03,817 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:55285 2023-03-22 17:54:03,818 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:03,818 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,818 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:55285 2023-03-22 17:54:03,818 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:03,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:03,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:03,825 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:03,825 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,825 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,825 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,826 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:55285 2023-03-22 17:54:03,826 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:03,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:03,829 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:03,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:03,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:03,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:03,829 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:03,830 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,830 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,830 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:03,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:03,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:03,838 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:03,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:03,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:03,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:03,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:03,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:03,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:03,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:03,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:03,839 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,839 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:03,839 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:03,839 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:03,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:03,839 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:03,939 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:03,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/1bf6640a5909456d99195e5ad268862c 2023-03-22 17:54:04,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/1b78de980cda4106b4b573b3a4e22760 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1b78de980cda4106b4b573b3a4e22760 2023-03-22 17:54:04,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1b78de980cda4106b4b573b3a4e22760, entries=150, sequenceid=1030, filesize=11.9 K 2023-03-22 17:54:04,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/a0eb626f9b5f46e58fea6fc7777c01e8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a0eb626f9b5f46e58fea6fc7777c01e8 2023-03-22 17:54:04,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a0eb626f9b5f46e58fea6fc7777c01e8, entries=150, sequenceid=1030, filesize=11.9 K 2023-03-22 17:54:04,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/1bf6640a5909456d99195e5ad268862c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1bf6640a5909456d99195e5ad268862c 2023-03-22 17:54:04,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1bf6640a5909456d99195e5ad268862c, entries=150, sequenceid=1030, filesize=11.9 K 2023-03-22 17:54:04,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.78 KB/136995 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 883ms, sequenceid=1030, compaction requested=true 2023-03-22 17:54:04,018 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:04,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:04,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:04,018 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:04,019 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:04,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:04,019 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:04,020 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/A is initiating minor compaction (all files) 2023-03-22 17:54:04,020 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:04,020 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/A in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:04,020 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/B is initiating minor compaction (all files) 2023-03-22 17:54:04,020 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/0679bfb3d6d349469fb2b747c5bf4655, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a5b1c0609817428b83a7750a93c8b22f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1b78de980cda4106b4b573b3a4e22760] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=31.9 K 2023-03-22 17:54:04,020 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/B in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:04,020 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/c93fea7fc5fc40849da909dfbe9e74bd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a22374b639424bfdb7f80db3f1121660, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a0eb626f9b5f46e58fea6fc7777c01e8] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=31.9 K 2023-03-22 17:54:04,020 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0679bfb3d6d349469fb2b747c5bf4655, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1679507641007 2023-03-22 17:54:04,020 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting c93fea7fc5fc40849da909dfbe9e74bd, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1679507641007 2023-03-22 17:54:04,020 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a5b1c0609817428b83a7750a93c8b22f, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=1016, earliestPutTs=1679507641296 2023-03-22 17:54:04,020 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a22374b639424bfdb7f80db3f1121660, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=1016, earliestPutTs=1679507641296 2023-03-22 17:54:04,021 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1b78de980cda4106b4b573b3a4e22760, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1679507643131 2023-03-22 17:54:04,021 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a0eb626f9b5f46e58fea6fc7777c01e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1679507643131 2023-03-22 17:54:04,032 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#A#compaction#99 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:04,033 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#B#compaction#100 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:04,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:54:04,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=140.49 KB heapSize=404.70 KB 2023-03-22 17:54:04,270 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63859c62 to 127.0.0.1:55285 2023-03-22 17:54:04,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:54:04,270 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:04,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:54:04,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:54:04,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:54:04,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:54:04,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:54:04,316 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61e72c30 to 127.0.0.1:55285 2023-03-22 17:54:04,316 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:04,445 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/c36c1a708b534303ae100a416d322207 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/c36c1a708b534303ae100a416d322207 2023-03-22 17:54:04,445 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/1c69cad1fa58434792859fd5e8ca5016 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1c69cad1fa58434792859fd5e8ca5016 2023-03-22 17:54:04,450 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/A of 00ab0c7e213b0d56501a0c20a5e0b9c6 into 1c69cad1fa58434792859fd5e8ca5016(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:04,450 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/B of 00ab0c7e213b0d56501a0c20a5e0b9c6 into c36c1a708b534303ae100a416d322207(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:04,450 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:04,451 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:04,451 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/A, priority=13, startTime=1679507644018; duration=0sec 2023-03-22 17:54:04,451 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/B, priority=13, startTime=1679507644018; duration=0sec 2023-03-22 17:54:04,451 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:04,451 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:04,451 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:04,452 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:04,452 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 00ab0c7e213b0d56501a0c20a5e0b9c6/C is initiating minor compaction (all files) 2023-03-22 17:54:04,452 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ab0c7e213b0d56501a0c20a5e0b9c6/C in TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:04,452 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/aed902e6db2c4fefbcadfa0c0dded275, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/f235b982cf6e4d49a4d5382f5288a834, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1bf6640a5909456d99195e5ad268862c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp, totalSize=31.9 K 2023-03-22 17:54:04,452 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting aed902e6db2c4fefbcadfa0c0dded275, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1010, earliestPutTs=1679507641007 2023-03-22 17:54:04,453 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f235b982cf6e4d49a4d5382f5288a834, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=1016, earliestPutTs=1679507641296 2023-03-22 17:54:04,453 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1bf6640a5909456d99195e5ad268862c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1679507643131 2023-03-22 17:54:04,459 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ab0c7e213b0d56501a0c20a5e0b9c6#C#compaction#102 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:04,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1068 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/76bc95d6b9d34c87b61f04d2f9846a05 2023-03-22 17:54:04,876 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/cae60c86b9434ca7b1b0674ec050723a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/cae60c86b9434ca7b1b0674ec050723a 2023-03-22 17:54:04,881 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ab0c7e213b0d56501a0c20a5e0b9c6/C of 00ab0c7e213b0d56501a0c20a5e0b9c6 into cae60c86b9434ca7b1b0674ec050723a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:04,881 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:04,881 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6., storeName=00ab0c7e213b0d56501a0c20a5e0b9c6/C, priority=13, startTime=1679507644018; duration=0sec 2023-03-22 17:54:04,881 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:05,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1068 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/267d41c9b2a0402a82f8011145295446 2023-03-22 17:54:05,425 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:55285 2023-03-22 17:54:05,425 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:05,445 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:55285 2023-03-22 17:54:05,445 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:05,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=1068 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/400e571748db4cc2aa8e24b2998443c8 2023-03-22 17:54:05,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/76bc95d6b9d34c87b61f04d2f9846a05 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/76bc95d6b9d34c87b61f04d2f9846a05 2023-03-22 17:54:05,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/76bc95d6b9d34c87b61f04d2f9846a05, entries=150, sequenceid=1068, filesize=11.9 K 2023-03-22 17:54:05,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/267d41c9b2a0402a82f8011145295446 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/267d41c9b2a0402a82f8011145295446 2023-03-22 17:54:05,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/267d41c9b2a0402a82f8011145295446, entries=150, sequenceid=1068, filesize=11.9 K 2023-03-22 17:54:05,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/400e571748db4cc2aa8e24b2998443c8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/400e571748db4cc2aa8e24b2998443c8 2023-03-22 17:54:05,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/400e571748db4cc2aa8e24b2998443c8, entries=150, sequenceid=1068, filesize=11.9 K 2023-03-22 17:54:05,524 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.49 KB/143865, heapSize ~404.65 KB/414360, currentSize=20.13 KB/20610 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 1254ms, sequenceid=1068, compaction requested=false 2023-03-22 17:54:05,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:08,781 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:55285 2023-03-22 17:54:08,781 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:08,781 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 340 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 145 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 166 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 303 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 4961 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 4662 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 2059 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 6170 rows 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 2045 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 6134 rows 2023-03-22 17:54:08,782 INFO [Listener at localhost/44491] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-22 17:54:08,782 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44ef43e3 to 127.0.0.1:55285 2023-03-22 17:54:08,782 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:08,787 INFO [Listener at localhost/44491] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-22 17:54:08,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-22 17:54:08,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:08,801 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507648801"}]},"ts":"1679507648801"} 2023-03-22 17:54:08,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-22 17:54:08,802 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-22 17:54:08,805 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-22 17:54:08,807 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ab0c7e213b0d56501a0c20a5e0b9c6, UNASSIGN}] 2023-03-22 17:54:08,807 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ab0c7e213b0d56501a0c20a5e0b9c6, UNASSIGN 2023-03-22 17:54:08,808 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=00ab0c7e213b0d56501a0c20a5e0b9c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:08,808 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507648808"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507648808"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507648808"}]},"ts":"1679507648808"} 2023-03-22 17:54:08,809 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:54:08,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-22 17:54:08,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:54:08,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 00ab0c7e213b0d56501a0c20a5e0b9c6, disabling compactions & flushes 2023-03-22 17:54:08,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:08,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:08,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. after waiting 0 ms 2023-03-22 17:54:08,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:08,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 00ab0c7e213b0d56501a0c20a5e0b9c6 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-22 17:54:08,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=A 2023-03-22 17:54:08,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:08,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=B 2023-03-22 17:54:08,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:08,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ab0c7e213b0d56501a0c20a5e0b9c6, store=C 2023-03-22 17:54:08,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:09,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-22 17:54:09,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1078 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/a28b28ce6beb410f9ad243934af3fc91 2023-03-22 17:54:09,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-22 17:54:09,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1078 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/19c79a3eeccb4d0ea9b7b7ffeb9ce3e5 2023-03-22 17:54:09,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-22 17:54:10,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1078 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/a33651424de34b6689d4347233005b71 2023-03-22 17:54:10,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/A/a28b28ce6beb410f9ad243934af3fc91 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a28b28ce6beb410f9ad243934af3fc91 2023-03-22 17:54:10,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a28b28ce6beb410f9ad243934af3fc91, entries=100, sequenceid=1078, filesize=9.5 K 2023-03-22 17:54:10,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/B/19c79a3eeccb4d0ea9b7b7ffeb9ce3e5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/19c79a3eeccb4d0ea9b7b7ffeb9ce3e5 2023-03-22 17:54:10,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/19c79a3eeccb4d0ea9b7b7ffeb9ce3e5, entries=100, sequenceid=1078, filesize=9.5 K 2023-03-22 17:54:10,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/.tmp/C/a33651424de34b6689d4347233005b71 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a33651424de34b6689d4347233005b71 2023-03-22 17:54:10,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a33651424de34b6689d4347233005b71, entries=100, sequenceid=1078, filesize=9.5 K 2023-03-22 17:54:10,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 00ab0c7e213b0d56501a0c20a5e0b9c6 in 1252ms, sequenceid=1078, compaction requested=true 2023-03-22 17:54:10,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7f8d3f8eadc240b8b731b84694bd83c6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1211ad3865b34110999773e9edb03ca3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/604f49fc72794b769c2a8020fa66acfb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/6e9a01f31dd04d26994e22fcc12621c2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a9c0239a3b6141b0bd7e6e135d55fabf, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/c756756fe017416cbe8902994db95c1a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/de3019ca34944aa8a3bd1f1c7dab3135, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5b4b2e4d6fa348c0849dd332d5afd68b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/bb1a1c4a87324cbfa8a0753769d9bc1a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/01f1c849cc2d4ec78c35d9647d1f2a19, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/64ba1774b43c4ad488d2458663bb87f3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/44a1b2fe804244e1991a898a5fe78c9d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/59fa985310e545918e1d4514de8d8dc6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f11e0d5a83e0441c972b46b4c3545dd9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/19cea5577f47437d94040440c70fd427, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/ccfaccbe0d754795bc0dab6f06086bc8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/daacd7cba00745a194e17c9ed13e2345, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/237efc1e44934537aa20fd641db32ddc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/35337279be4044118e1a6be6a4f27b14, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5830f3124e0444d48887c06aa2723048, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/dea26a033b99477db4dbac7b72c28d3f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/54f6552cb91d444080768d0940a1f1a7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a581c2356e6a410e9be965a1d9a7fd29, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/943be75650f848bea8329a2ad27f85ed, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/24bd830bd7504a18b89ba162cba57850, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/18795c57df0b4591b7144f3f60840401, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/438880eed8b34806b07ef79304ca115e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f19951dea1de4696851c33781da64eff, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7c4587a07c8640e19eadd63d0a9c8ddc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/0679bfb3d6d349469fb2b747c5bf4655, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1eb0de01b71e4d839bb7edafb65f5539, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a5b1c0609817428b83a7750a93c8b22f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1b78de980cda4106b4b573b3a4e22760] to archive 2023-03-22 17:54:10,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:54:10,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7f8d3f8eadc240b8b731b84694bd83c6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7f8d3f8eadc240b8b731b84694bd83c6 2023-03-22 17:54:10,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1211ad3865b34110999773e9edb03ca3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1211ad3865b34110999773e9edb03ca3 2023-03-22 17:54:10,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/604f49fc72794b769c2a8020fa66acfb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/604f49fc72794b769c2a8020fa66acfb 2023-03-22 17:54:10,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/6e9a01f31dd04d26994e22fcc12621c2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/6e9a01f31dd04d26994e22fcc12621c2 2023-03-22 17:54:10,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a9c0239a3b6141b0bd7e6e135d55fabf to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a9c0239a3b6141b0bd7e6e135d55fabf 2023-03-22 17:54:10,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/c756756fe017416cbe8902994db95c1a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/c756756fe017416cbe8902994db95c1a 2023-03-22 17:54:10,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/de3019ca34944aa8a3bd1f1c7dab3135 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/de3019ca34944aa8a3bd1f1c7dab3135 2023-03-22 17:54:10,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5b4b2e4d6fa348c0849dd332d5afd68b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5b4b2e4d6fa348c0849dd332d5afd68b 2023-03-22 17:54:10,257 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/bb1a1c4a87324cbfa8a0753769d9bc1a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/bb1a1c4a87324cbfa8a0753769d9bc1a 2023-03-22 17:54:10,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/01f1c849cc2d4ec78c35d9647d1f2a19 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/01f1c849cc2d4ec78c35d9647d1f2a19 2023-03-22 17:54:10,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/64ba1774b43c4ad488d2458663bb87f3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/64ba1774b43c4ad488d2458663bb87f3 2023-03-22 17:54:10,261 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/44a1b2fe804244e1991a898a5fe78c9d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/44a1b2fe804244e1991a898a5fe78c9d 2023-03-22 17:54:10,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/59fa985310e545918e1d4514de8d8dc6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/59fa985310e545918e1d4514de8d8dc6 2023-03-22 17:54:10,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f11e0d5a83e0441c972b46b4c3545dd9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f11e0d5a83e0441c972b46b4c3545dd9 2023-03-22 17:54:10,264 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/19cea5577f47437d94040440c70fd427 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/19cea5577f47437d94040440c70fd427 2023-03-22 17:54:10,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/ccfaccbe0d754795bc0dab6f06086bc8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/ccfaccbe0d754795bc0dab6f06086bc8 2023-03-22 17:54:10,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/daacd7cba00745a194e17c9ed13e2345 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/daacd7cba00745a194e17c9ed13e2345 2023-03-22 17:54:10,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/237efc1e44934537aa20fd641db32ddc to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/237efc1e44934537aa20fd641db32ddc 2023-03-22 17:54:10,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/35337279be4044118e1a6be6a4f27b14 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/35337279be4044118e1a6be6a4f27b14 2023-03-22 17:54:10,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5830f3124e0444d48887c06aa2723048 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/5830f3124e0444d48887c06aa2723048 2023-03-22 17:54:10,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/dea26a033b99477db4dbac7b72c28d3f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/dea26a033b99477db4dbac7b72c28d3f 2023-03-22 17:54:10,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/54f6552cb91d444080768d0940a1f1a7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/54f6552cb91d444080768d0940a1f1a7 2023-03-22 17:54:10,274 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a581c2356e6a410e9be965a1d9a7fd29 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a581c2356e6a410e9be965a1d9a7fd29 2023-03-22 17:54:10,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/943be75650f848bea8329a2ad27f85ed to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/943be75650f848bea8329a2ad27f85ed 2023-03-22 17:54:10,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/24bd830bd7504a18b89ba162cba57850 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/24bd830bd7504a18b89ba162cba57850 2023-03-22 17:54:10,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/18795c57df0b4591b7144f3f60840401 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/18795c57df0b4591b7144f3f60840401 2023-03-22 17:54:10,278 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/438880eed8b34806b07ef79304ca115e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/438880eed8b34806b07ef79304ca115e 2023-03-22 17:54:10,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f19951dea1de4696851c33781da64eff to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/f19951dea1de4696851c33781da64eff 2023-03-22 17:54:10,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7c4587a07c8640e19eadd63d0a9c8ddc to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/7c4587a07c8640e19eadd63d0a9c8ddc 2023-03-22 17:54:10,281 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/0679bfb3d6d349469fb2b747c5bf4655 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/0679bfb3d6d349469fb2b747c5bf4655 2023-03-22 17:54:10,282 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1eb0de01b71e4d839bb7edafb65f5539 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1eb0de01b71e4d839bb7edafb65f5539 2023-03-22 17:54:10,284 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a5b1c0609817428b83a7750a93c8b22f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a5b1c0609817428b83a7750a93c8b22f 2023-03-22 17:54:10,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1b78de980cda4106b4b573b3a4e22760 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1b78de980cda4106b4b573b3a4e22760 2023-03-22 17:54:10,309 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4ce32607e9cd400aa3dc7f1b3829aa74, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4a95cdda7ef346a3a9d7d8c9d611d1ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/bc8db7073f5848f5a094696418ee3919, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/60549289b32647ad90e156f379b72366, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3d653c9ff3fe4455a4943fec1aa1caea, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1951dec72a244256a60278c912683606, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1d4228a536d9472a986ebce9cc8eeedb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/d98cbb8eed514263afd67ff22df983c3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a7805e04f0824294921a32fb06e099e6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b37a291cdf6434d8a9a1de718d062fa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/fe2959318c75483fbd8ba1036405c4f4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/704c8ae87f6c4391acbb4265fcfc008c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/0be1d79c5785449b8c9be4321e859989, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b0f183c8f0746449e167bb0e1226ea0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1c34e5831e8c4430873fd81e900f9153, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/08dad7ee6d7a4716babb2802e6761314, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e9407018377147ebbfde5a97ab140e55, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/b3cbf39a4a974dd894aac654e161c0a2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/429b55bde83a4741864c75e7cbc6126b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4e3a317e29694e39b5ba4253942358a6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/9abcfc68cb2e4dc1984e9562ec737910, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f1171554ade644d48d384e9cfd3143ef, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e17b86b27c394df8b12c2054936c09db, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/478f17600c7540e58d902f0129ce9381, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/93605b870d0044398df72fe9702fe8dd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f577e021bfe44057bb7474ade816282b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1faeaaafb64641d9acf4b49533b90b19, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/2c45b2f3890244bdada917465a0935e4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/ed3b1ce1af3f4b36ae25f60500228c0a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/c93fea7fc5fc40849da909dfbe9e74bd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/716885e7816545da8b124e48eeea20fd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a22374b639424bfdb7f80db3f1121660, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a0eb626f9b5f46e58fea6fc7777c01e8] to archive 2023-03-22 17:54:10,310 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:54:10,312 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4ce32607e9cd400aa3dc7f1b3829aa74 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4ce32607e9cd400aa3dc7f1b3829aa74 2023-03-22 17:54:10,313 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4a95cdda7ef346a3a9d7d8c9d611d1ae to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4a95cdda7ef346a3a9d7d8c9d611d1ae 2023-03-22 17:54:10,315 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/bc8db7073f5848f5a094696418ee3919 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/bc8db7073f5848f5a094696418ee3919 2023-03-22 17:54:10,316 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/60549289b32647ad90e156f379b72366 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/60549289b32647ad90e156f379b72366 2023-03-22 17:54:10,317 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3d653c9ff3fe4455a4943fec1aa1caea to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3d653c9ff3fe4455a4943fec1aa1caea 2023-03-22 17:54:10,318 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1951dec72a244256a60278c912683606 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1951dec72a244256a60278c912683606 2023-03-22 17:54:10,319 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1d4228a536d9472a986ebce9cc8eeedb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1d4228a536d9472a986ebce9cc8eeedb 2023-03-22 17:54:10,321 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/d98cbb8eed514263afd67ff22df983c3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/d98cbb8eed514263afd67ff22df983c3 2023-03-22 17:54:10,322 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a7805e04f0824294921a32fb06e099e6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a7805e04f0824294921a32fb06e099e6 2023-03-22 17:54:10,323 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b37a291cdf6434d8a9a1de718d062fa to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b37a291cdf6434d8a9a1de718d062fa 2023-03-22 17:54:10,324 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/fe2959318c75483fbd8ba1036405c4f4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/fe2959318c75483fbd8ba1036405c4f4 2023-03-22 17:54:10,325 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/704c8ae87f6c4391acbb4265fcfc008c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/704c8ae87f6c4391acbb4265fcfc008c 2023-03-22 17:54:10,326 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/0be1d79c5785449b8c9be4321e859989 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/0be1d79c5785449b8c9be4321e859989 2023-03-22 17:54:10,328 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b0f183c8f0746449e167bb0e1226ea0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/3b0f183c8f0746449e167bb0e1226ea0 2023-03-22 17:54:10,329 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1c34e5831e8c4430873fd81e900f9153 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1c34e5831e8c4430873fd81e900f9153 2023-03-22 17:54:10,330 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/08dad7ee6d7a4716babb2802e6761314 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/08dad7ee6d7a4716babb2802e6761314 2023-03-22 17:54:10,331 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e9407018377147ebbfde5a97ab140e55 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e9407018377147ebbfde5a97ab140e55 2023-03-22 17:54:10,332 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/b3cbf39a4a974dd894aac654e161c0a2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/b3cbf39a4a974dd894aac654e161c0a2 2023-03-22 17:54:10,333 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/429b55bde83a4741864c75e7cbc6126b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/429b55bde83a4741864c75e7cbc6126b 2023-03-22 17:54:10,335 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4e3a317e29694e39b5ba4253942358a6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/4e3a317e29694e39b5ba4253942358a6 2023-03-22 17:54:10,336 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/9abcfc68cb2e4dc1984e9562ec737910 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/9abcfc68cb2e4dc1984e9562ec737910 2023-03-22 17:54:10,337 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f1171554ade644d48d384e9cfd3143ef to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f1171554ade644d48d384e9cfd3143ef 2023-03-22 17:54:10,338 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e17b86b27c394df8b12c2054936c09db to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/e17b86b27c394df8b12c2054936c09db 2023-03-22 17:54:10,339 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/478f17600c7540e58d902f0129ce9381 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/478f17600c7540e58d902f0129ce9381 2023-03-22 17:54:10,340 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/93605b870d0044398df72fe9702fe8dd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/93605b870d0044398df72fe9702fe8dd 2023-03-22 17:54:10,341 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f577e021bfe44057bb7474ade816282b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/f577e021bfe44057bb7474ade816282b 2023-03-22 17:54:10,342 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1faeaaafb64641d9acf4b49533b90b19 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/1faeaaafb64641d9acf4b49533b90b19 2023-03-22 17:54:10,343 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/2c45b2f3890244bdada917465a0935e4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/2c45b2f3890244bdada917465a0935e4 2023-03-22 17:54:10,343 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/ed3b1ce1af3f4b36ae25f60500228c0a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/ed3b1ce1af3f4b36ae25f60500228c0a 2023-03-22 17:54:10,345 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/c93fea7fc5fc40849da909dfbe9e74bd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/c93fea7fc5fc40849da909dfbe9e74bd 2023-03-22 17:54:10,346 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/716885e7816545da8b124e48eeea20fd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/716885e7816545da8b124e48eeea20fd 2023-03-22 17:54:10,347 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a22374b639424bfdb7f80db3f1121660 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a22374b639424bfdb7f80db3f1121660 2023-03-22 17:54:10,348 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a0eb626f9b5f46e58fea6fc7777c01e8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/a0eb626f9b5f46e58fea6fc7777c01e8 2023-03-22 17:54:10,352 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b566ab03a3f742c893524ae180b46a3d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b0582734ba5d498fbf1d25eae2eb0c6f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1b857a0f0153429ab94faf0e8a510924, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/35e71bfb1f9349a2830bd2a4f3f80739, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/dd3281f0e8f04332808f0f41f076fa4e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1c0f87e6a9fb4e18856396f059ee6cb0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d49fd1b223c1407e9239bcf7569cde59, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8b8a81f308744dc2aea2f817a7aca222, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/4644b931ce0a4107a26c5baa9897d557, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/29574ce7014f41159cd82fe86fb77fa1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3dff0829169344ed8b7397c2831e3a8c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/fc527653ed8843bd9e5767dda96150ec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ad4ac79fc2a84527a832c4c91b7720eb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d208ef6d2ed0403abc6276536c941a03, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/c99dde605dcc42c7a81893157d79301a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ed5d50ad6ec740129a5172652e197edc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/076c51e348f04ff1b529203dbace9d4a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/2b09f8a3a83248e2b259551f8bc851e1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3bcbfbea38e0489aa19505f818647347, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/6cd11b2f3a1e434f922f1c2057a63801, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a60076075f59461ab05036dabb15377b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/feedb777e5934374be208d698af281d6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/0f6f6103adb54c85a74bffb5a5938065, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/69abe2351c5f465c964610067cf85bdf, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ac91e00f0c5749d8ba63eb2eb9abea66, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/e77179c2b24a43ac8d3edaff1e8a7540, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ce59b2b16bb64e0f9162a4ec862408fa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/7f5871e21069424a909b71c797f51eca, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/02e88216bd8a4bf3ae686da8ed744dba, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/aed902e6db2c4fefbcadfa0c0dded275, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8fa6b7e494084270bc496c4a95277548, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/f235b982cf6e4d49a4d5382f5288a834, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1bf6640a5909456d99195e5ad268862c] to archive 2023-03-22 17:54:10,353 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:54:10,354 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b566ab03a3f742c893524ae180b46a3d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b566ab03a3f742c893524ae180b46a3d 2023-03-22 17:54:10,355 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b0582734ba5d498fbf1d25eae2eb0c6f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/b0582734ba5d498fbf1d25eae2eb0c6f 2023-03-22 17:54:10,356 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1b857a0f0153429ab94faf0e8a510924 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1b857a0f0153429ab94faf0e8a510924 2023-03-22 17:54:10,357 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/35e71bfb1f9349a2830bd2a4f3f80739 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/35e71bfb1f9349a2830bd2a4f3f80739 2023-03-22 17:54:10,358 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/dd3281f0e8f04332808f0f41f076fa4e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/dd3281f0e8f04332808f0f41f076fa4e 2023-03-22 17:54:10,359 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1c0f87e6a9fb4e18856396f059ee6cb0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1c0f87e6a9fb4e18856396f059ee6cb0 2023-03-22 17:54:10,360 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d49fd1b223c1407e9239bcf7569cde59 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d49fd1b223c1407e9239bcf7569cde59 2023-03-22 17:54:10,362 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8b8a81f308744dc2aea2f817a7aca222 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8b8a81f308744dc2aea2f817a7aca222 2023-03-22 17:54:10,363 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/4644b931ce0a4107a26c5baa9897d557 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/4644b931ce0a4107a26c5baa9897d557 2023-03-22 17:54:10,364 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/29574ce7014f41159cd82fe86fb77fa1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/29574ce7014f41159cd82fe86fb77fa1 2023-03-22 17:54:10,366 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3dff0829169344ed8b7397c2831e3a8c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3dff0829169344ed8b7397c2831e3a8c 2023-03-22 17:54:10,367 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/fc527653ed8843bd9e5767dda96150ec to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/fc527653ed8843bd9e5767dda96150ec 2023-03-22 17:54:10,369 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ad4ac79fc2a84527a832c4c91b7720eb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ad4ac79fc2a84527a832c4c91b7720eb 2023-03-22 17:54:10,370 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d208ef6d2ed0403abc6276536c941a03 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/d208ef6d2ed0403abc6276536c941a03 2023-03-22 17:54:10,371 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/c99dde605dcc42c7a81893157d79301a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/c99dde605dcc42c7a81893157d79301a 2023-03-22 17:54:10,372 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ed5d50ad6ec740129a5172652e197edc to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ed5d50ad6ec740129a5172652e197edc 2023-03-22 17:54:10,374 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/076c51e348f04ff1b529203dbace9d4a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/076c51e348f04ff1b529203dbace9d4a 2023-03-22 17:54:10,375 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/2b09f8a3a83248e2b259551f8bc851e1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/2b09f8a3a83248e2b259551f8bc851e1 2023-03-22 17:54:10,376 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3bcbfbea38e0489aa19505f818647347 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/3bcbfbea38e0489aa19505f818647347 2023-03-22 17:54:10,377 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/6cd11b2f3a1e434f922f1c2057a63801 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/6cd11b2f3a1e434f922f1c2057a63801 2023-03-22 17:54:10,378 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a60076075f59461ab05036dabb15377b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a60076075f59461ab05036dabb15377b 2023-03-22 17:54:10,379 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/feedb777e5934374be208d698af281d6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/feedb777e5934374be208d698af281d6 2023-03-22 17:54:10,380 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/0f6f6103adb54c85a74bffb5a5938065 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/0f6f6103adb54c85a74bffb5a5938065 2023-03-22 17:54:10,381 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/69abe2351c5f465c964610067cf85bdf to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/69abe2351c5f465c964610067cf85bdf 2023-03-22 17:54:10,382 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ac91e00f0c5749d8ba63eb2eb9abea66 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ac91e00f0c5749d8ba63eb2eb9abea66 2023-03-22 17:54:10,383 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/e77179c2b24a43ac8d3edaff1e8a7540 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/e77179c2b24a43ac8d3edaff1e8a7540 2023-03-22 17:54:10,385 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ce59b2b16bb64e0f9162a4ec862408fa to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/ce59b2b16bb64e0f9162a4ec862408fa 2023-03-22 17:54:10,386 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/7f5871e21069424a909b71c797f51eca to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/7f5871e21069424a909b71c797f51eca 2023-03-22 17:54:10,387 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/02e88216bd8a4bf3ae686da8ed744dba to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/02e88216bd8a4bf3ae686da8ed744dba 2023-03-22 17:54:10,388 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/aed902e6db2c4fefbcadfa0c0dded275 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/aed902e6db2c4fefbcadfa0c0dded275 2023-03-22 17:54:10,390 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8fa6b7e494084270bc496c4a95277548 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/8fa6b7e494084270bc496c4a95277548 2023-03-22 17:54:10,391 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/f235b982cf6e4d49a4d5382f5288a834 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/f235b982cf6e4d49a4d5382f5288a834 2023-03-22 17:54:10,392 DEBUG [StoreCloser-TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1bf6640a5909456d99195e5ad268862c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/1bf6640a5909456d99195e5ad268862c 2023-03-22 17:54:10,397 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/recovered.edits/1081.seqid, newMaxSeqId=1081, maxSeqId=1 2023-03-22 17:54:10,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6. 2023-03-22 17:54:10,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 00ab0c7e213b0d56501a0c20a5e0b9c6: 2023-03-22 17:54:10,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:54:10,401 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=00ab0c7e213b0d56501a0c20a5e0b9c6, regionState=CLOSED 2023-03-22 17:54:10,401 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507650401"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507650401"}]},"ts":"1679507650401"} 2023-03-22 17:54:10,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-22 17:54:10,405 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 00ab0c7e213b0d56501a0c20a5e0b9c6, server=jenkins-hbase4.apache.org,34403,1679507617723 in 1.5940 sec 2023-03-22 17:54:10,406 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-22 17:54:10,407 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ab0c7e213b0d56501a0c20a5e0b9c6, UNASSIGN in 1.5990 sec 2023-03-22 17:54:10,407 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507650407"}]},"ts":"1679507650407"} 2023-03-22 17:54:10,408 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-22 17:54:10,410 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-22 17:54:10,412 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6180 sec 2023-03-22 17:54:10,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-22 17:54:10,906 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-22 17:54:10,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-22 17:54:10,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:10,918 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:10,920 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:10,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-22 17:54:10,924 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:54:10,928 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/recovered.edits] 2023-03-22 17:54:10,932 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1c69cad1fa58434792859fd5e8ca5016 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/1c69cad1fa58434792859fd5e8ca5016 2023-03-22 17:54:10,933 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/76bc95d6b9d34c87b61f04d2f9846a05 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/76bc95d6b9d34c87b61f04d2f9846a05 2023-03-22 17:54:10,934 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a28b28ce6beb410f9ad243934af3fc91 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/A/a28b28ce6beb410f9ad243934af3fc91 2023-03-22 17:54:10,937 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/19c79a3eeccb4d0ea9b7b7ffeb9ce3e5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/19c79a3eeccb4d0ea9b7b7ffeb9ce3e5 2023-03-22 17:54:10,938 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/267d41c9b2a0402a82f8011145295446 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/267d41c9b2a0402a82f8011145295446 2023-03-22 17:54:10,939 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/c36c1a708b534303ae100a416d322207 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/B/c36c1a708b534303ae100a416d322207 2023-03-22 17:54:10,942 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/400e571748db4cc2aa8e24b2998443c8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/400e571748db4cc2aa8e24b2998443c8 2023-03-22 17:54:10,943 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a33651424de34b6689d4347233005b71 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/a33651424de34b6689d4347233005b71 2023-03-22 17:54:10,945 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/cae60c86b9434ca7b1b0674ec050723a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/C/cae60c86b9434ca7b1b0674ec050723a 2023-03-22 17:54:10,947 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/recovered.edits/1081.seqid to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6/recovered.edits/1081.seqid 2023-03-22 17:54:10,948 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/00ab0c7e213b0d56501a0c20a5e0b9c6 2023-03-22 17:54:10,948 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:54:10,950 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:10,977 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-22 17:54:11,019 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-22 17:54:11,021 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:11,021 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-22 17:54:11,021 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679507651021"}]},"ts":"9223372036854775807"} 2023-03-22 17:54:11,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-22 17:54:11,023 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-22 17:54:11,023 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 00ab0c7e213b0d56501a0c20a5e0b9c6, NAME => 'TestAcidGuarantees,,1679507621571.00ab0c7e213b0d56501a0c20a5e0b9c6.', STARTKEY => '', ENDKEY => ''}] 2023-03-22 17:54:11,023 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-22 17:54:11,024 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679507651023"}]},"ts":"9223372036854775807"} 2023-03-22 17:54:11,025 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-22 17:54:11,027 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:11,029 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 115 msec 2023-03-22 17:54:11,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-22 17:54:11,224 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-22 17:54:11,241 INFO [Listener at localhost/44491] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=231 (was 205) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2a4b69b2-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:34403-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-582541700_15 at /127.0.0.1:60616 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2a4b69b2-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2a4b69b2-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2a4b69b2-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=545 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=278 (was 282), ProcessCount=169 (was 169), AvailableMemoryMB=6164 (was 6720) 2023-03-22 17:54:11,251 INFO [Listener at localhost/44491] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=278, ProcessCount=169, AvailableMemoryMB=6164 2023-03-22 17:54:11,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:54:11,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:54:11,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:11,256 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-22 17:54:11,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-22 17:54:11,257 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-22 17:54:11,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-22 17:54:11,259 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:11,259 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b empty. 2023-03-22 17:54:11,260 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:11,260 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:54:11,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-22 17:54:11,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-22 17:54:11,668 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:54:11,670 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 44f11d39aca56af66573ce1ce740420b, NAME => 'TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp 2023-03-22 17:54:11,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-22 17:54:12,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:54:12,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 44f11d39aca56af66573ce1ce740420b, disabling compactions & flushes 2023-03-22 17:54:12,077 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:12,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:12,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. after waiting 0 ms 2023-03-22 17:54:12,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:12,077 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:12,077 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:12,080 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-22 17:54:12,080 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507652080"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507652080"}]},"ts":"1679507652080"} 2023-03-22 17:54:12,082 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-22 17:54:12,082 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-22 17:54:12,083 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507652082"}]},"ts":"1679507652082"} 2023-03-22 17:54:12,083 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-22 17:54:12,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, ASSIGN}] 2023-03-22 17:54:12,089 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, ASSIGN 2023-03-22 17:54:12,089 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=false 2023-03-22 17:54:12,240 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=44f11d39aca56af66573ce1ce740420b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:12,241 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507652240"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507652240"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507652240"}]},"ts":"1679507652240"} 2023-03-22 17:54:12,242 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:54:12,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-22 17:54:12,398 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:12,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 44f11d39aca56af66573ce1ce740420b, NAME => 'TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:54:12,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:54:12,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,400 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,401 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A 2023-03-22 17:54:12,401 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A 2023-03-22 17:54:12,401 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:12,402 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44f11d39aca56af66573ce1ce740420b columnFamilyName A 2023-03-22 17:54:12,402 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(310): Store=44f11d39aca56af66573ce1ce740420b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:12,402 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,403 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B 2023-03-22 17:54:12,403 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B 2023-03-22 17:54:12,404 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:12,404 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44f11d39aca56af66573ce1ce740420b columnFamilyName B 2023-03-22 17:54:12,404 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(310): Store=44f11d39aca56af66573ce1ce740420b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:12,404 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,405 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C 2023-03-22 17:54:12,405 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C 2023-03-22 17:54:12,406 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:12,406 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44f11d39aca56af66573ce1ce740420b columnFamilyName C 2023-03-22 17:54:12,406 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(310): Store=44f11d39aca56af66573ce1ce740420b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:12,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:12,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:54:12,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:12,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:54:12,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 44f11d39aca56af66573ce1ce740420b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71171699, jitterRate=0.06054095923900604}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:54:12,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:12,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., pid=18, masterSystemTime=1679507652394 2023-03-22 17:54:12,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:12,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:12,416 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=44f11d39aca56af66573ce1ce740420b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:12,416 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507652415"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507652415"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507652415"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507652415"}]},"ts":"1679507652415"} 2023-03-22 17:54:12,419 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-22 17:54:12,419 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 in 175 msec 2023-03-22 17:54:12,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-22 17:54:12,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, ASSIGN in 332 msec 2023-03-22 17:54:12,421 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-22 17:54:12,421 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507652421"}]},"ts":"1679507652421"} 2023-03-22 17:54:12,422 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-22 17:54:12,424 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-22 17:54:12,425 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-03-22 17:54:13,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-22 17:54:13,363 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-22 17:54:13,364 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x33c943f0 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:13,369 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b8d0eb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:13,370 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:13,374 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40350, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:13,375 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-22 17:54:13,378 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42010, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-22 17:54:13,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:54:13,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:54:13,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:13,806 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-22 17:54:13,807 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:54:13,807 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-22 17:54:13,812 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-22 17:54:13,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, REOPEN/MOVE}] 2023-03-22 17:54:13,824 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, REOPEN/MOVE 2023-03-22 17:54:13,825 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=44f11d39aca56af66573ce1ce740420b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:13,825 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507653825"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507653825"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507653825"}]},"ts":"1679507653825"} 2023-03-22 17:54:13,826 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:54:13,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:13,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 44f11d39aca56af66573ce1ce740420b, disabling compactions & flushes 2023-03-22 17:54:13,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:13,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:13,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. after waiting 0 ms 2023-03-22 17:54:13,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:13,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-22 17:54:13,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:13,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:13,984 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 44f11d39aca56af66573ce1ce740420b to self. 2023-03-22 17:54:13,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:13,986 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=44f11d39aca56af66573ce1ce740420b, regionState=CLOSED 2023-03-22 17:54:13,986 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507653986"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507653986"}]},"ts":"1679507653986"} 2023-03-22 17:54:13,988 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-22 17:54:13,989 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 in 161 msec 2023-03-22 17:54:13,989 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=true 2023-03-22 17:54:14,139 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=44f11d39aca56af66573ce1ce740420b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,140 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507654139"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507654139"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507654139"}]},"ts":"1679507654139"} 2023-03-22 17:54:14,142 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:54:14,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 44f11d39aca56af66573ce1ce740420b, NAME => 'TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:54:14,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:54:14,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,299 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,300 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A 2023-03-22 17:54:14,300 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A 2023-03-22 17:54:14,300 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:14,303 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44f11d39aca56af66573ce1ce740420b columnFamilyName A 2023-03-22 17:54:14,305 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(310): Store=44f11d39aca56af66573ce1ce740420b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:14,305 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,306 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B 2023-03-22 17:54:14,306 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B 2023-03-22 17:54:14,306 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:14,307 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44f11d39aca56af66573ce1ce740420b columnFamilyName B 2023-03-22 17:54:14,307 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(310): Store=44f11d39aca56af66573ce1ce740420b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:14,307 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,308 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C 2023-03-22 17:54:14,308 DEBUG [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C 2023-03-22 17:54:14,308 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:14,309 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44f11d39aca56af66573ce1ce740420b columnFamilyName C 2023-03-22 17:54:14,309 INFO [StoreOpener-44f11d39aca56af66573ce1ce740420b-1] regionserver.HStore(310): Store=44f11d39aca56af66573ce1ce740420b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:14,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:54:14,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 44f11d39aca56af66573ce1ce740420b; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62886995, jitterRate=-0.06291075050830841}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:54:14,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:14,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., pid=23, masterSystemTime=1679507654293 2023-03-22 17:54:14,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,317 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=44f11d39aca56af66573ce1ce740420b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,318 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507654317"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507654317"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507654317"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507654317"}]},"ts":"1679507654317"} 2023-03-22 17:54:14,320 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-22 17:54:14,320 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 in 177 msec 2023-03-22 17:54:14,321 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-22 17:54:14,321 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, REOPEN/MOVE in 496 msec 2023-03-22 17:54:14,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-22 17:54:14,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 512 msec 2023-03-22 17:54:14,326 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 936 msec 2023-03-22 17:54:14,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-22 17:54:14,334 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x5b3934fb to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,339 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34135972, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,340 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x1aa5eef5 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,345 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a71edf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,345 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,349 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,349 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x69f15cb4 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,354 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7aba8e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,354 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,358 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,358 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x54451f11 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,363 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c2598b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,363 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,367 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,367 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,370 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,371 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:14,374 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:14,380 DEBUG [hconnection-0x7abd27d9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:14,384 DEBUG [hconnection-0x5bf8eed2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,384 DEBUG [hconnection-0x5ab9a78c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,387 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40376, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,388 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40360, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,389 DEBUG [hconnection-0x1696913d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,390 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40390, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,390 DEBUG [hconnection-0x6fa38319-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,390 DEBUG [hconnection-0x46262b6e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,391 DEBUG [hconnection-0x1ec56119-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,390 DEBUG [hconnection-0x6b438383-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,405 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40400, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,405 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40380, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,407 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40410, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,405 DEBUG [hconnection-0x51285737-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:14,403 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40418, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:14,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:14,409 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:14,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:14,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:14,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:14,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,416 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40444, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,417 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40428, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:14,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,424 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:14,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:14,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:14,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:14,442 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-22 17:54:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:14,442 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:14,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:14,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:14,444 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:14,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:14,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:14,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:14,451 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:14,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:14,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36e835eb[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:14,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:14,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,456 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:14,457 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,457 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:14,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:14,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:14,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:14,464 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:14,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:14,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:14,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:14,464 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:14,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,471 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507714467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507714467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507714468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507714467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:14,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:14,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:14,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:14,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:14,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:14,476 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:14,476 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:14,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:14,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507714473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507714574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507714574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507714575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,576 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:14,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:14,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507714579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507714579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:14,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:14,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:14,596 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:14,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:14,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:14,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:14,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,599 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:14,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:14,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:14,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:14,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:14,601 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:14,601 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:14,601 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:14,604 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,604 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:14,604 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:14,604 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:14,609 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa032b8[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:14,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:14,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:14,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:14,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:14,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:14,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:14,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:14,617 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:14,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:14,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:14,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:14,617 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:14,619 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:14,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:14,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:14,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:14,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:14,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:14,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:14,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:14,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:14,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:14,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:14,633 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:14,633 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:14,733 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:14,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:14,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:14,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:14,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:14,742 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:14,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:14,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:14,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:14,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,744 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:14,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:14,745 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,745 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:14,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:14,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:14,746 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:14,746 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:14,746 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:14,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:14,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:14,747 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,747 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:14,747 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:14,747 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:14,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:14,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:14,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:14,749 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:14,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515614e1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:14,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:14,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:14,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:14,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:14,754 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,754 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:14,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:14,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:14,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:14,759 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:14,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:14,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:14,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:14,759 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:14,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:14,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,762 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:14,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:14,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:14,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:14,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:14,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:14,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:14,769 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:14,769 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:14,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507714777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507714778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507714782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507714782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:14,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507714783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,869 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:14,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:14,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:14,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:14,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:14,890 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:14,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:14,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:14,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:14,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:14,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:14,892 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:14,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:14,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:14,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:14,896 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,896 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:14,897 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:14,897 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:14,898 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f64cf19[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:14,901 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:14,901 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,902 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,902 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:14,903 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,903 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:14,903 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:14,903 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:14,906 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:14,906 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:14,906 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:14,906 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:14,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:14,912 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:14,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:14,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:14,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:14,912 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:14,913 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322e3608406c0154aac93ec946d743ab9e3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e3608406c0154aac93ec946d743ab9e3 2023-03-22 17:54:14,916 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,916 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:14,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:14,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:14,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:14,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:14,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:14,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:14,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:14,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:14,924 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:14,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:14,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:14,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:14,925 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:14,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:14,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:14,925 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:14,925 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:14,934 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/9b5f89b5d09140fa878a64fbc8f8c322 2023-03-22 17:54:14,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/fca4bfb0375349d9a3c9ac686abdbd04 2023-03-22 17:54:15,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/b1d2ddc5f90a443bb9615a46c76f7525 2023-03-22 17:54:15,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/9b5f89b5d09140fa878a64fbc8f8c322 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/9b5f89b5d09140fa878a64fbc8f8c322 2023-03-22 17:54:15,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/9b5f89b5d09140fa878a64fbc8f8c322, entries=250, sequenceid=18, filesize=38.7 K 2023-03-22 17:54:15,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/fca4bfb0375349d9a3c9ac686abdbd04 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fca4bfb0375349d9a3c9ac686abdbd04 2023-03-22 17:54:15,025 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:15,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:15,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:15,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:15,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:15,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fca4bfb0375349d9a3c9ac686abdbd04, entries=150, sequenceid=18, filesize=11.6 K 2023-03-22 17:54:15,030 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:15,030 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,030 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:15,030 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:15,030 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,030 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/b1d2ddc5f90a443bb9615a46c76f7525 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/b1d2ddc5f90a443bb9615a46c76f7525 2023-03-22 17:54:15,032 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,032 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,032 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,032 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:15,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:15,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:15,033 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:15,033 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:15,034 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,034 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:15,034 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:15,034 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:15,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/b1d2ddc5f90a443bb9615a46c76f7525, entries=150, sequenceid=18, filesize=11.6 K 2023-03-22 17:54:15,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 44f11d39aca56af66573ce1ce740420b in 594ms, sequenceid=18, compaction requested=false 2023-03-22 17:54:15,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:15,037 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,037 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,037 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,037 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:15,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1854e873[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:15,041 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:15,041 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,042 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:15,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,045 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:15,045 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:15,045 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:15,046 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,046 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:15,046 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,046 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:15,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:15,051 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:15,051 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:15,051 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:15,051 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:15,051 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:15,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,069 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:15,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:15,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:15,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,070 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:15,070 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:15,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:15,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-22 17:54:15,082 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:15,082 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:15,082 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:15,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507715095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507715098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507715098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507715102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507715103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,115 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322aa113818cf104f87828bcf3e308c2e16 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322aa113818cf104f87828bcf3e308c2e16 2023-03-22 17:54:15,170 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:15,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:15,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:15,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:15,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:15,181 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:15,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:15,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:15,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,183 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:15,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:15,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:15,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:15,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:15,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:15,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:15,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:15,187 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:15,189 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:15,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff2c33[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:15,189 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:15,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:15,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:15,192 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,192 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:15,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,194 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,194 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,194 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:15,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:15,197 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:15,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:15,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:15,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:15,197 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:15,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,201 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,204 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:15,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,205 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:15,205 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:15,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507715204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507715204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507715205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507715205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507715206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,305 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:15,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:15,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:15,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:15,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:15,308 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:15,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:15,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:15,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,310 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:15,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:15,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:15,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:15,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:15,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:15,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:15,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,313 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:15,313 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:15,313 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:15,316 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,316 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:15,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67efd111[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:15,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:15,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:15,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:15,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:15,320 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,320 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:15,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:15,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:15,324 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:15,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:15,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:15,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:15,324 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:15,327 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,328 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:15,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:15,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,331 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,331 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:15,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,331 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:15,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507715407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507715407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507715408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507715409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507715410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,432 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:15,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:15,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:15,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:15,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:15,449 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:15,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:15,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:15,450 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,450 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,452 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,452 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,453 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,453 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:15,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:15,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:15,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:15,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:15,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:15,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:15,469 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:15,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:15,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:15,472 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:15,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ebac6f2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:15,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:15,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:15,478 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:15,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:15,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,482 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:15,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:15,485 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:15,485 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:15,485 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:15,485 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:15,485 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:15,488 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,488 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,488 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,494 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:15,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:15,494 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,495 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:15,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:15,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,495 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:15,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,541 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/f96105752e624eaaa473f0fe0afb1b79 2023-03-22 17:54:15,595 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:15,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:15,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:15,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:15,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:15,599 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:15,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:15,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:15,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:15,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:15,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:15,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:15,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:15,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:15,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:15,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:15,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:15,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1112a426[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:15,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:15,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:15,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:15,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:15,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:15,619 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:15,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:15,622 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:15,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:15,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:15,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:15,622 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:15,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,627 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:15,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:15,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,633 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:15,633 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:15,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507715710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507715710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507715711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507715716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:15,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507715717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,733 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:15,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:15,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:15,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:15,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:15,740 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:15,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:15,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:15,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,745 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,745 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,745 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:15,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:15,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:15,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:15,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:15,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:15,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:15,750 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,750 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:15,750 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:15,751 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:15,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:15,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:15,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:15,752 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ef39c83[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:15,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:15,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:15,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:15,760 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,760 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:15,763 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:15,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:15,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:15,766 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:15,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:15,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:15,767 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:15,767 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:15,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,770 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,773 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,776 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:15,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,777 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:15,777 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:15,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,877 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:15,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:15,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:15,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:15,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:15,883 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:15,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:15,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:15,884 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,884 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,885 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:15,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:15,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:15,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:15,892 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:15,892 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:15,892 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:15,893 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,893 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:15,893 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:15,893 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,894 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:15,894 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,897 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:15,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@778dbf15[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:15,900 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:15,900 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,904 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,904 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:15,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:15,905 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:15,905 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:15,906 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:15,906 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:15,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:15,910 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:15,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:15,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:15,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:15,910 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:15,912 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,912 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,912 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:15,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,913 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,916 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:15,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:15,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:15,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:15,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:15,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:15,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:15,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:15,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:15,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:15,921 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:15,921 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:15,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:15,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:15,950 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-22 17:54:15,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/bea102982edf44ad82fdbea4ba31fa53 2023-03-22 17:54:16,014 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/6355064a01934b9aa9680bdab9beefc2 2023-03-22 17:54:16,021 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:16,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:16,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/f96105752e624eaaa473f0fe0afb1b79 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f96105752e624eaaa473f0fe0afb1b79 2023-03-22 17:54:16,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:16,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:16,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:16,028 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:16,028 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:16,028 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:16,028 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:16,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f96105752e624eaaa473f0fe0afb1b79, entries=200, sequenceid=37, filesize=31.9 K 2023-03-22 17:54:16,029 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,029 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:16,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/bea102982edf44ad82fdbea4ba31fa53 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/bea102982edf44ad82fdbea4ba31fa53 2023-03-22 17:54:16,030 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:16,030 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:16,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:16,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:16,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:16,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/bea102982edf44ad82fdbea4ba31fa53, entries=150, sequenceid=37, filesize=11.6 K 2023-03-22 17:54:16,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/6355064a01934b9aa9680bdab9beefc2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6355064a01934b9aa9680bdab9beefc2 2023-03-22 17:54:16,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:16,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:16,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:16,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:16,041 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:16,041 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:16,041 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:16,042 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:16,042 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:16,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6355064a01934b9aa9680bdab9beefc2, entries=150, sequenceid=37, filesize=11.6 K 2023-03-22 17:54:16,044 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:16,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 44f11d39aca56af66573ce1ce740420b in 962ms, sequenceid=37, compaction requested=false 2023-03-22 17:54:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:16,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff87a40[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:16,046 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:16,046 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:16,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:16,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:16,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:16,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:16,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:16,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:16,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:16,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:16,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:16,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:16,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:16,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:16,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:16,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:16,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:16,058 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:16,059 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:16,059 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:16,059 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:16,059 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:16,061 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,061 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,061 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:16,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:16,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:16,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:16,066 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:16,066 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:16,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:16,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:16,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:16,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:16,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:16,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:16,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:16,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:16,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,068 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:16,068 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:16,067 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:16,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:16,168 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:16,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:16,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:16,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:16,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:16,172 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:16,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:16,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:16,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:16,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:16,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:16,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:16,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:16,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:16,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:16,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:16,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:16,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:16,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:16,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:16,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:16,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:16,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:16,175 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:16,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:16,176 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-22 17:54:16,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:16,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:16,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:16,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:16,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:16,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:16,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:16,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:16,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507716227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507716229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507716231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507716233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507716233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507716334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507716334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507716337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507716337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507716340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507716538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507716541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507716544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507716545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507716546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,594 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303229db2f2cae98345379ae626ed93d9a7ea to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303229db2f2cae98345379ae626ed93d9a7ea 2023-03-22 17:54:16,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507716841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507716846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507716847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507716848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:16,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:16,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507716848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,000 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/cb3ee42ddc8d42d392f56b1ddd088557 2023-03-22 17:54:17,026 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/3e1dcda1b202406d8faf572379dd3f50 2023-03-22 17:54:17,036 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/c7b0e5c03925446184ee7ba61f11e75b 2023-03-22 17:54:17,041 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/cb3ee42ddc8d42d392f56b1ddd088557 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/cb3ee42ddc8d42d392f56b1ddd088557 2023-03-22 17:54:17,048 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/cb3ee42ddc8d42d392f56b1ddd088557, entries=150, sequenceid=51, filesize=25.1 K 2023-03-22 17:54:17,049 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/3e1dcda1b202406d8faf572379dd3f50 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3e1dcda1b202406d8faf572379dd3f50 2023-03-22 17:54:17,054 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3e1dcda1b202406d8faf572379dd3f50, entries=150, sequenceid=51, filesize=11.6 K 2023-03-22 17:54:17,055 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/c7b0e5c03925446184ee7ba61f11e75b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/c7b0e5c03925446184ee7ba61f11e75b 2023-03-22 17:54:17,062 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/c7b0e5c03925446184ee7ba61f11e75b, entries=150, sequenceid=51, filesize=11.6 K 2023-03-22 17:54:17,063 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 44f11d39aca56af66573ce1ce740420b in 887ms, sequenceid=51, compaction requested=true 2023-03-22 17:54:17,063 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:17,063 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:17,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:17,065 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:17,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:17,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bcfdc0d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:17,069 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:17,069 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:17,071 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:17,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:17,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:17,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:17,077 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:17,078 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:17,078 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:17,078 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:17,078 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:17,080 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,080 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:17,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,080 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:17,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:17,083 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,087 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,091 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:17,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:17,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:17,093 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:17,093 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:17,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:17,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:17,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:17,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:17,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:17,193 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:17,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:17,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:17,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:17,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:17,196 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:17,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:17,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:17,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:17,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:17,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:17,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:17,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:17,201 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:17,201 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:17,201 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:17,201 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:17,201 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:17,201 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:17,201 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:17,201 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-22 17:54:17,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:17,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:17,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:17,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:17,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:17,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:17,226 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322940aa05c92984500a49ff73a27d8c2fd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322940aa05c92984500a49ff73a27d8c2fd 2023-03-22 17:54:17,232 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/a274a97230dd47b6adcb6ff6d046875f 2023-03-22 17:54:17,245 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/28eee1e31b224575b4bb4ab117b0ce80 2023-03-22 17:54:17,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:17,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:17,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507717361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507717362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507717364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507717364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507717365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507717466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507717466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507717467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507717467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507717468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,657 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/1395644ef8d0482ca71fda83c85a57f0 2023-03-22 17:54:17,662 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/a274a97230dd47b6adcb6ff6d046875f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a274a97230dd47b6adcb6ff6d046875f 2023-03-22 17:54:17,666 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a274a97230dd47b6adcb6ff6d046875f, entries=150, sequenceid=69, filesize=25.1 K 2023-03-22 17:54:17,667 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/28eee1e31b224575b4bb4ab117b0ce80 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/28eee1e31b224575b4bb4ab117b0ce80 2023-03-22 17:54:17,671 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/28eee1e31b224575b4bb4ab117b0ce80, entries=150, sequenceid=69, filesize=11.6 K 2023-03-22 17:54:17,674 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/1395644ef8d0482ca71fda83c85a57f0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1395644ef8d0482ca71fda83c85a57f0 2023-03-22 17:54:17,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507717668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507717669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507717670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507717668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:17,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507717674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,683 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1395644ef8d0482ca71fda83c85a57f0, entries=150, sequenceid=69, filesize=11.6 K 2023-03-22 17:54:17,684 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 44f11d39aca56af66573ce1ce740420b in 483ms, sequenceid=69, compaction requested=true 2023-03-22 17:54:17,684 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:17,684 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:17,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:17,686 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:17,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:17,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:17,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79aa4e73[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:17,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:17,689 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,690 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,690 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:17,690 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:17,690 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:17,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:17,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:17,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:17,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:17,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:17,694 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:17,694 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:17,694 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:17,694 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:17,694 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:17,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:17,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:17,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:17,698 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,699 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:17,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:17,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:17,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:17,702 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:17,702 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:17,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:17,802 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:17,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:17,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:17,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:17,812 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:17,812 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:17,812 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:17,812 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:17,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,814 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:17,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:17,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:17,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:17,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:17,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:17,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:17,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:17,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:17,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:17,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:17,816 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:17,816 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:17,816 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-22 17:54:17,816 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:17,817 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:17,817 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:17,817 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:17,817 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:17,817 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:17,844 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322c86ac56f3b1c421cb20b2ccb9ae89976 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c86ac56f3b1c421cb20b2ccb9ae89976 2023-03-22 17:54:17,855 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/487416a1b17546ea922cee1bbacf7c88 2023-03-22 17:54:17,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:17,983 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:18,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507717997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507718000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507718000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507718001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507718001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507718102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507718107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507718107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507718108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507718108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,277 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/e40c4e6ea62243ada670c1e49b9b6377 2023-03-22 17:54:18,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507718308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507718309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507718310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507718310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507718313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507718611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507718613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507718614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507718614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:18,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507718617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,690 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/532248871c5f4145b6b28dc59c2815f3 2023-03-22 17:54:18,696 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/487416a1b17546ea922cee1bbacf7c88 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/487416a1b17546ea922cee1bbacf7c88 2023-03-22 17:54:18,702 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/487416a1b17546ea922cee1bbacf7c88, entries=150, sequenceid=83, filesize=25.1 K 2023-03-22 17:54:18,703 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/e40c4e6ea62243ada670c1e49b9b6377 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/e40c4e6ea62243ada670c1e49b9b6377 2023-03-22 17:54:18,713 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/e40c4e6ea62243ada670c1e49b9b6377, entries=150, sequenceid=83, filesize=11.6 K 2023-03-22 17:54:18,714 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/532248871c5f4145b6b28dc59c2815f3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/532248871c5f4145b6b28dc59c2815f3 2023-03-22 17:54:18,718 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/532248871c5f4145b6b28dc59c2815f3, entries=150, sequenceid=83, filesize=11.6 K 2023-03-22 17:54:18,719 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 44f11d39aca56af66573ce1ce740420b in 903ms, sequenceid=83, compaction requested=true 2023-03-22 17:54:18,719 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:18,719 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:18,719 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:18,719 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:18,730 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,730 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:18,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63452344[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:18,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:18,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,733 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:18,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,734 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:18,735 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:18,737 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:18,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:18,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:18,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:18,737 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:18,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:18,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,750 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,755 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:18,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:18,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:18,755 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:18,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,756 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:18,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:18,856 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:18,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:18,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:18,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:18,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:18,860 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:18,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:18,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:18,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:18,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:18,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:18,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:18,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:18,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:18,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:18,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:18,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:18,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:18,863 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:18,863 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:18,863 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-22 17:54:18,863 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:18,863 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:18,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:18,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:18,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:18,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:18,880 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322f543a0f38dfe4633bcf4a3f53ce0a42c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f543a0f38dfe4633bcf4a3f53ce0a42c 2023-03-22 17:54:18,893 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/22bfc8b330564b098ecc81104cee75df 2023-03-22 17:54:18,910 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/b6ff0b3852124325af927db1ff57ef31 2023-03-22 17:54:18,928 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/44221fe2a5bd4934be7700edf276ef64 2023-03-22 17:54:18,933 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/22bfc8b330564b098ecc81104cee75df as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/22bfc8b330564b098ecc81104cee75df 2023-03-22 17:54:18,939 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/22bfc8b330564b098ecc81104cee75df, entries=150, sequenceid=102, filesize=25.1 K 2023-03-22 17:54:18,940 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/b6ff0b3852124325af927db1ff57ef31 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b6ff0b3852124325af927db1ff57ef31 2023-03-22 17:54:18,950 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b6ff0b3852124325af927db1ff57ef31, entries=150, sequenceid=102, filesize=11.6 K 2023-03-22 17:54:18,951 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/44221fe2a5bd4934be7700edf276ef64 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/44221fe2a5bd4934be7700edf276ef64 2023-03-22 17:54:18,962 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/44221fe2a5bd4934be7700edf276ef64, entries=150, sequenceid=102, filesize=11.6 K 2023-03-22 17:54:18,963 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 44f11d39aca56af66573ce1ce740420b in 100ms, sequenceid=102, compaction requested=true 2023-03-22 17:54:18,963 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:18,963 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:18,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:18,967 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fa36f16[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:18,969 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:18,969 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,970 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,971 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:18,971 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:18,971 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:18,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:18,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:18,975 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:18,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:18,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:18,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:18,975 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:18,977 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,977 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,977 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:18,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:18,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:18,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:18,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:18,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:18,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:18,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:18,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:18,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:18,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:18,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:18,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:18,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:18,989 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:18,989 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:19,089 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:19,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:19,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:19,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:19,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:19,116 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:19,116 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,116 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:19,116 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:19,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,118 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,118 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:19,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,122 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:19,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:19,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:19,123 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,123 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:19,124 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:19,124 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:19,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,125 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af9f15b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:19,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:19,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:19,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:19,133 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,133 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,133 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,133 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:19,136 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:19,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:19,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:19,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:19,136 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:19,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,139 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,140 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:54:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:19,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:19,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:19,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:19,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:19,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,143 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,143 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:19,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:19,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:19,144 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,145 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:19,145 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:19,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679507659142, max timestamp=1679507659195Num uniques 859; 2023-03-22 17:54:19,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-22 17:54:19,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679507659142, max timestamp=1679507659195Num uniques 859; 2023-03-22 17:54:19,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-22 17:54:19,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679507659142, max timestamp=1679507659195Num uniques 859; 2023-03-22 17:54:19,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-22 17:54:19,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507719199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507719199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507719200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507719203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507719202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,245 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:19,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:19,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:19,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:19,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:19,249 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:19,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:19,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:19,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,251 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,252 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,252 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:19,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:19,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:19,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:19,260 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,260 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:19,260 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:19,260 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:19,261 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:19,261 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,261 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:19,261 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:19,263 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@710bf5a1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:19,266 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:19,266 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,267 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:19,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:19,268 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,268 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:19,269 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:19,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:19,272 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:19,272 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:19,272 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:19,272 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:19,272 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:19,278 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,278 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,278 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,282 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,288 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,288 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,288 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,288 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:19,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,288 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,288 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,289 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:19,289 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:19,289 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,289 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,289 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,289 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507719312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507719312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507719312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507719312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507719312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,389 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:19,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:19,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:19,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:19,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:19,393 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:19,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:19,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:19,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:19,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:19,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:19,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:19,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:19,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:19,403 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:19,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b9ea7c7[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:19,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:19,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,406 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:19,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:19,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:19,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:19,408 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:19,410 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:19,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:19,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:19,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:19,410 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:19,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,413 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,414 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,417 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:19,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:19,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:19,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,418 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:19,418 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,418 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:19,418 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507719515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,518 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:19,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507719515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507719517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507719517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507719517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:19,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:19,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:19,522 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:19,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:19,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:19,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:19,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:19,526 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,527 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:19,527 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:19,527 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,527 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:19,527 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:19,527 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,527 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:19,527 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:19,529 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:19,531 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:19,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f71b47d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:19,531 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:19,532 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:19,533 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,533 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:19,535 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,535 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,535 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,535 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:19,538 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:19,538 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:19,538 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:19,538 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:19,538 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:19,540 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,540 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,540 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,541 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,543 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:19,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:19,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:19,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,548 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:19,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,549 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:19,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,565 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032288c6bd2ab44343b4a11a451638315687 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032288c6bd2ab44343b4a11a451638315687 2023-03-22 17:54:19,649 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:19,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:19,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:19,652 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:19,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:19,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:19,653 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,653 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,655 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,655 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,656 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:19,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:19,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:19,657 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,657 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:19,657 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:19,657 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:19,659 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:19,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:19,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a97ffe1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:19,662 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:19,663 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,663 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:19,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:19,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:19,664 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:19,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:19,670 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:19,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:19,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:19,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:19,670 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:19,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,674 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,675 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:19,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,681 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:19,681 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:19,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,781 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:19,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:19,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:19,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:19,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:19,785 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:19,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:19,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:19,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,790 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:19,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:19,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:19,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:19,792 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,792 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:19,792 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:19,792 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:19,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:19,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,794 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:19,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341fa7ef[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:19,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:19,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:19,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:19,799 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:19,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:19,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:19,801 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:19,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:19,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:19,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:19,801 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:19,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,804 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,807 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,807 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,807 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:19,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:19,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,808 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,808 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,808 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,808 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,808 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:19,808 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,808 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,808 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,808 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:19,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:19,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507719820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507719821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507719821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507719822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507719822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,908 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:19,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:19,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:19,912 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:19,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:19,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:19,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,912 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,913 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,914 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,914 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,914 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:19,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:19,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:19,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:19,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:19,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:19,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:19,915 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,916 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:19,916 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:19,916 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:19,916 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:19,916 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:19,916 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,916 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:19,916 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:19,919 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:19,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7af19e34[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:19,919 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:19,922 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,922 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:19,923 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:19,925 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:19,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:19,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:19,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:19,925 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:19,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:19,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:19,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:19,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:19,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:19,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:19,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:19,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:19,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:19,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:19,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:19,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:19,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:19,931 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:19,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:19,931 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:19,971 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=114, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/f536b79cd4924a7a8b1349e971c0c37f 2023-03-22 17:54:20,031 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:20,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:20,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:20,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:20,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:20,039 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:20,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:20,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:20,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:20,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,041 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:20,041 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,043 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,043 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:20,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:20,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:20,055 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,055 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:20,056 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:20,056 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:20,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:20,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78211f4b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:20,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:20,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,062 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:20,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:20,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:20,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:20,069 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:20,069 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:20,069 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:20,069 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:20,069 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:20,072 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,072 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:20,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,072 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,075 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:20,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:20,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:20,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:20,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:20,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:20,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:20,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:20,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,079 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:20,079 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:20,179 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:20,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:20,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:20,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:20,183 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:20,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:20,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:20,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:20,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:20,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:20,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:20,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:20,187 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,187 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:20,187 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:20,187 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:20,192 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:20,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dec8f59[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:20,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:20,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:20,195 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,195 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:20,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,196 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:20,196 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,196 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:20,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:20,198 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:20,198 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:20,198 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:20,198 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:20,198 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:20,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:20,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:20,202 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,203 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:20,205 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:20,205 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:20,205 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:20,306 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:20,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:20,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:20,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:20,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:20,309 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:20,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:20,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:20,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:20,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:20,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:20,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:20,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:20,313 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,313 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:20,313 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:20,313 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:20,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,315 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,315 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,315 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:20,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:20,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14056bd4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:20,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:20,319 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:20,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,319 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:20,320 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:20,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:20,322 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:20,323 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:20,323 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:20,323 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:20,323 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:20,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:20,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:20,325 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,326 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:20,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507720325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:20,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507720326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,328 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:20,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:20,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,329 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:20,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:20,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:20,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:20,329 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:20,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:20,329 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:20,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:20,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507720328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:20,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507720329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:20,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507720329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/c9a9662fe7b04f5b804f1072590ad0b2 2023-03-22 17:54:20,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/2ab6131ecee74e99adae1479d1f2343e 2023-03-22 17:54:20,400 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/f536b79cd4924a7a8b1349e971c0c37f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f536b79cd4924a7a8b1349e971c0c37f 2023-03-22 17:54:20,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f536b79cd4924a7a8b1349e971c0c37f, entries=400, sequenceid=114, filesize=59.0 K 2023-03-22 17:54:20,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/c9a9662fe7b04f5b804f1072590ad0b2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/c9a9662fe7b04f5b804f1072590ad0b2 2023-03-22 17:54:20,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/c9a9662fe7b04f5b804f1072590ad0b2, entries=150, sequenceid=114, filesize=11.6 K 2023-03-22 17:54:20,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/2ab6131ecee74e99adae1479d1f2343e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/2ab6131ecee74e99adae1479d1f2343e 2023-03-22 17:54:20,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/2ab6131ecee74e99adae1479d1f2343e, entries=150, sequenceid=114, filesize=11.6 K 2023-03-22 17:54:20,417 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 44f11d39aca56af66573ce1ce740420b in 1277ms, sequenceid=114, compaction requested=true 2023-03-22 17:54:20,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:20,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:20,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:20,417 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-22 17:54:20,417 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-22 17:54:20,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:20,419 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-22 17:54:20,420 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/B is initiating minor compaction (all files) 2023-03-22 17:54:20,420 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 235754 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-22 17:54:20,420 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/B in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,420 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/A is initiating minor compaction (all files) 2023-03-22 17:54:20,420 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/A in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,420 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fca4bfb0375349d9a3c9ac686abdbd04, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/bea102982edf44ad82fdbea4ba31fa53, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3e1dcda1b202406d8faf572379dd3f50, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/28eee1e31b224575b4bb4ab117b0ce80, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/e40c4e6ea62243ada670c1e49b9b6377, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b6ff0b3852124325af927db1ff57ef31, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/c9a9662fe7b04f5b804f1072590ad0b2] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=81.3 K 2023-03-22 17:54:20,420 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/9b5f89b5d09140fa878a64fbc8f8c322, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f96105752e624eaaa473f0fe0afb1b79, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/cb3ee42ddc8d42d392f56b1ddd088557, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a274a97230dd47b6adcb6ff6d046875f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/487416a1b17546ea922cee1bbacf7c88, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/22bfc8b330564b098ecc81104cee75df, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f536b79cd4924a7a8b1349e971c0c37f] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=230.2 K 2023-03-22 17:54:20,421 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting fca4bfb0375349d9a3c9ac686abdbd04, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679507654428 2023-03-22 17:54:20,424 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting bea102982edf44ad82fdbea4ba31fa53, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679507654465 2023-03-22 17:54:20,424 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 9b5f89b5d09140fa878a64fbc8f8c322, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679507654428 2023-03-22 17:54:20,425 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3e1dcda1b202406d8faf572379dd3f50, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507655091 2023-03-22 17:54:20,425 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f96105752e624eaaa473f0fe0afb1b79, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679507654465 2023-03-22 17:54:20,425 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 28eee1e31b224575b4bb4ab117b0ce80, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679507656230 2023-03-22 17:54:20,425 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting cb3ee42ddc8d42d392f56b1ddd088557, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507655091 2023-03-22 17:54:20,426 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e40c4e6ea62243ada670c1e49b9b6377, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679507657362 2023-03-22 17:54:20,426 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a274a97230dd47b6adcb6ff6d046875f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679507656230 2023-03-22 17:54:20,426 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b6ff0b3852124325af927db1ff57ef31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679507657998 2023-03-22 17:54:20,427 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 487416a1b17546ea922cee1bbacf7c88, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679507657362 2023-03-22 17:54:20,427 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting c9a9662fe7b04f5b804f1072590ad0b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507659137 2023-03-22 17:54:20,427 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 22bfc8b330564b098ecc81104cee75df, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679507657998 2023-03-22 17:54:20,428 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f536b79cd4924a7a8b1349e971c0c37f, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507659125 2023-03-22 17:54:20,430 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:20,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:20,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:20,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:20,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:20,451 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:20,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:20,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:20,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:20,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:20,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:20,455 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#B#compaction#130 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:20,456 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:20,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:20,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:20,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:20,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:20,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:20,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:20,470 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-03-22 17:54:20,471 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:20,471 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:20,471 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:20,471 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:20,471 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:20,471 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:20,481 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#A#compaction#129 average throughput is 0.68 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:20,490 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/2273bbd0bb794a2bacf9c3702aaae1c6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/2273bbd0bb794a2bacf9c3702aaae1c6 2023-03-22 17:54:20,501 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/B of 44f11d39aca56af66573ce1ce740420b into 2273bbd0bb794a2bacf9c3702aaae1c6(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:20,501 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:20,501 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/B, priority=9, startTime=1679507660417; duration=0sec 2023-03-22 17:54:20,501 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:20,501 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-22 17:54:20,507 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-22 17:54:20,507 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/C is initiating minor compaction (all files) 2023-03-22 17:54:20,507 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/C in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:20,508 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/b1d2ddc5f90a443bb9615a46c76f7525, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6355064a01934b9aa9680bdab9beefc2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/c7b0e5c03925446184ee7ba61f11e75b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1395644ef8d0482ca71fda83c85a57f0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/532248871c5f4145b6b28dc59c2815f3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/44221fe2a5bd4934be7700edf276ef64, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/2ab6131ecee74e99adae1479d1f2343e] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=81.3 K 2023-03-22 17:54:20,509 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b1d2ddc5f90a443bb9615a46c76f7525, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679507654428 2023-03-22 17:54:20,510 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6355064a01934b9aa9680bdab9beefc2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679507654465 2023-03-22 17:54:20,510 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting c7b0e5c03925446184ee7ba61f11e75b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507655091 2023-03-22 17:54:20,511 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1395644ef8d0482ca71fda83c85a57f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679507656230 2023-03-22 17:54:20,511 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 532248871c5f4145b6b28dc59c2815f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679507657362 2023-03-22 17:54:20,512 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 44221fe2a5bd4934be7700edf276ef64, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679507657998 2023-03-22 17:54:20,513 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2ab6131ecee74e99adae1479d1f2343e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507659137 2023-03-22 17:54:20,533 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#C#compaction#132 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:20,539 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322c895344d78d349eea800426886a501fe to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c895344d78d349eea800426886a501fe 2023-03-22 17:54:20,540 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/73e76b0842904b49b2db95c3fd26ee76 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/73e76b0842904b49b2db95c3fd26ee76 2023-03-22 17:54:20,550 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/A of 44f11d39aca56af66573ce1ce740420b into 73e76b0842904b49b2db95c3fd26ee76(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:20,550 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:20,550 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/A, priority=9, startTime=1679507660417; duration=0sec 2023-03-22 17:54:20,550 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:20,556 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/58c7ced653174608aabe6579b8f01346 2023-03-22 17:54:20,558 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/122e97675054449ab56e466b1c21bda3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/122e97675054449ab56e466b1c21bda3 2023-03-22 17:54:20,566 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/C of 44f11d39aca56af66573ce1ce740420b into 122e97675054449ab56e466b1c21bda3(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:20,567 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:20,567 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/C, priority=9, startTime=1679507660417; duration=0sec 2023-03-22 17:54:20,567 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:20,581 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/7ce956c0211a40ef87b57c9ed00ae8a7 2023-03-22 17:54:20,994 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/4572c1fd3184423d84e0891d7b10cfa1 2023-03-22 17:54:20,999 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/58c7ced653174608aabe6579b8f01346 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/58c7ced653174608aabe6579b8f01346 2023-03-22 17:54:21,003 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/58c7ced653174608aabe6579b8f01346, entries=150, sequenceid=139, filesize=25.3 K 2023-03-22 17:54:21,005 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/7ce956c0211a40ef87b57c9ed00ae8a7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7ce956c0211a40ef87b57c9ed00ae8a7 2023-03-22 17:54:21,014 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7ce956c0211a40ef87b57c9ed00ae8a7, entries=150, sequenceid=139, filesize=11.8 K 2023-03-22 17:54:21,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/4572c1fd3184423d84e0891d7b10cfa1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/4572c1fd3184423d84e0891d7b10cfa1 2023-03-22 17:54:21,021 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/4572c1fd3184423d84e0891d7b10cfa1, entries=150, sequenceid=139, filesize=11.8 K 2023-03-22 17:54:21,022 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=0 B/0 for 44f11d39aca56af66573ce1ce740420b in 552ms, sequenceid=139, compaction requested=false 2023-03-22 17:54:21,022 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:21,022 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:21,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:21,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62f3575e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:21,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:21,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:21,027 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,028 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:21,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:21,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:21,031 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:21,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:21,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:21,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:21,031 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:21,034 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,034 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,034 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,041 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,043 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,044 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,044 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,044 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,044 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,044 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,044 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:21,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,044 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,047 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:21,047 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:21,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,147 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:21,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:21,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:21,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:21,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:21,151 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:21,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:21,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:21,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,153 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,153 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,153 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,153 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:21,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:21,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:21,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:21,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:21,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:21,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:21,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1258d70c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:21,159 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:21,159 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,165 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:21,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:21,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:21,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:21,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:21,170 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:21,170 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:21,170 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:21,170 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:21,170 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:21,172 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,172 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,172 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,178 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,178 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,178 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,178 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:21,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:21,177 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:21,178 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:21,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,178 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:21,278 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:21,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:21,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:21,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:21,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:21,283 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:21,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:21,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:21,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,285 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,285 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,286 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:21,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:21,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:21,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:21,288 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:21,288 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,289 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:21,289 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:21,289 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:21,291 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,291 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b771459[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:21,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:21,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,295 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:21,296 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:21,296 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:21,296 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,296 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:21,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,302 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:21,302 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,303 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:21,307 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:21,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:21,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:21,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:21,307 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:21,309 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,309 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,310 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,319 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:21,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:21,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:21,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,319 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:21,319 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:21,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:21,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:54:21,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:21,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:21,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:21,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:21,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:21,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:21,398 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322c5d7120d998348c2a8564e8448184751 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c5d7120d998348c2a8564e8448184751 2023-03-22 17:54:21,427 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:21,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:21,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:21,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:21,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:21,438 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:21,438 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,438 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:21,438 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:21,439 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,439 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39334, totalHeapSize=89576, min timestamp=1679507661374, max timestamp=1679507661437Num uniques 859; 2023-03-22 17:54:21,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=39334, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-22 17:54:21,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39334, totalHeapSize=89576, min timestamp=1679507661374, max timestamp=1679507661437Num uniques 859; 2023-03-22 17:54:21,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=39334, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-22 17:54:21,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39334, totalHeapSize=89576, min timestamp=1679507661374, max timestamp=1679507661437Num uniques 859; 2023-03-22 17:54:21,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=39334, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-22 17:54:21,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:21,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:21,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:21,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:21,445 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:21,445 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,445 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:21,449 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,449 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:21,449 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:21,449 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:21,454 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507721442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507721444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:21,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507721442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507721452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507721454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:21,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14138128[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:21,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:21,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:21,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:21,460 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,460 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:21,461 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:21,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:21,464 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:21,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:21,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:21,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:21,464 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:21,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,468 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,471 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:21,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:21,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,477 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,477 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,477 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,477 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,477 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:21,477 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:21,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507721556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507721556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507721557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507721557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507721557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,577 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:21,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:21,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:21,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:21,580 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:21,580 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:21,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:21,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:21,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:21,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:21,584 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,585 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:21,585 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:21,585 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:21,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5369ce6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:21,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:21,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:21,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,587 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,587 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:21,592 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:21,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:21,594 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:21,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:21,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:21,596 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:21,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:21,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:21,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:21,597 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:21,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,605 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,605 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,605 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,605 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:21,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:21,605 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,605 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:21,605 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,606 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:21,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,606 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:21,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,706 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:21,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:21,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:21,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:21,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:21,709 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:21,709 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,709 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:21,709 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:21,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:21,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:21,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:21,712 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,713 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:21,713 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:21,713 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:21,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:21,715 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,722 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,722 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:21,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c226d53[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:21,723 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:21,723 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,725 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,725 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:21,725 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:21,725 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:21,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:21,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,727 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:21,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:21,730 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:21,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:21,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:21,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:21,730 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:21,732 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,732 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,732 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,734 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:21,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:21,740 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:21,740 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:21,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:21,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507721760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507721761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507721763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507721763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:21,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507721763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,840 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:21,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:21,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:21,843 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/e8f5e7cec73c4c2c91e3d6fc4ace1412 2023-03-22 17:54:21,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:21,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:21,844 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:21,844 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,845 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:21,845 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:21,845 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,845 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,847 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:21,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:21,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:21,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:21,850 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,850 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:21,850 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:21,850 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:21,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,853 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,853 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:21,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3896376c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:21,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:21,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,856 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:21,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:21,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:21,856 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,856 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:21,858 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:21,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:21,862 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:21,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:21,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:21,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:21,863 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:21,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:21,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:21,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,870 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:21,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:21,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:21,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:21,871 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:21,871 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,871 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:21,871 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:21,871 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:21,871 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:21,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:21,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:21,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:21,871 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:21,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:21,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/5b1ccf8f3b1247f6bdf06d5354a8350c 2023-03-22 17:54:21,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/6cbb8c299a76404bbe2e5b3c457d5f16 2023-03-22 17:54:21,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/e8f5e7cec73c4c2c91e3d6fc4ace1412 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/e8f5e7cec73c4c2c91e3d6fc4ace1412 2023-03-22 17:54:21,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/e8f5e7cec73c4c2c91e3d6fc4ace1412, entries=400, sequenceid=155, filesize=59.4 K 2023-03-22 17:54:21,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/5b1ccf8f3b1247f6bdf06d5354a8350c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5b1ccf8f3b1247f6bdf06d5354a8350c 2023-03-22 17:54:21,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5b1ccf8f3b1247f6bdf06d5354a8350c, entries=150, sequenceid=155, filesize=11.8 K 2023-03-22 17:54:21,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/6cbb8c299a76404bbe2e5b3c457d5f16 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6cbb8c299a76404bbe2e5b3c457d5f16 2023-03-22 17:54:21,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6cbb8c299a76404bbe2e5b3c457d5f16, entries=150, sequenceid=155, filesize=11.8 K 2023-03-22 17:54:21,917 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 44f11d39aca56af66573ce1ce740420b in 547ms, sequenceid=155, compaction requested=true 2023-03-22 17:54:21,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:21,918 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:21,918 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:21,918 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:21,918 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:21,918 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:21,919 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 112736 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:21,919 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/A is initiating minor compaction (all files) 2023-03-22 17:54:21,919 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:21,919 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/A in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,919 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/B is initiating minor compaction (all files) 2023-03-22 17:54:21,919 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/B in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:21,919 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/73e76b0842904b49b2db95c3fd26ee76, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/58c7ced653174608aabe6579b8f01346, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/e8f5e7cec73c4c2c91e3d6fc4ace1412] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=110.1 K 2023-03-22 17:54:21,919 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/2273bbd0bb794a2bacf9c3702aaae1c6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7ce956c0211a40ef87b57c9ed00ae8a7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5b1ccf8f3b1247f6bdf06d5354a8350c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=35.3 K 2023-03-22 17:54:21,920 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 73e76b0842904b49b2db95c3fd26ee76, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507659137 2023-03-22 17:54:21,920 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2273bbd0bb794a2bacf9c3702aaae1c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507659137 2023-03-22 17:54:21,920 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 58c7ced653174608aabe6579b8f01346, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679507659197 2023-03-22 17:54:21,920 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7ce956c0211a40ef87b57c9ed00ae8a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679507659197 2023-03-22 17:54:21,921 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e8f5e7cec73c4c2c91e3d6fc4ace1412, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679507661361 2023-03-22 17:54:21,921 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 5b1ccf8f3b1247f6bdf06d5354a8350c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679507661368 2023-03-22 17:54:21,941 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#B#compaction#139 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:21,957 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#A#compaction#138 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:21,971 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:21,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:21,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:22,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:22,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:22,012 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:22,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:22,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:22,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:22,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:22,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:22,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:22,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:22,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:22,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:22,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:22,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:22,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:22,020 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:22,020 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:22,020 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:22,024 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:22,024 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:22,024 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-03-22 17:54:22,025 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:22,026 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:22,026 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:22,026 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:22,026 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:22,026 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:22,048 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/a6fc27e4cca640b59989489183324255 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a6fc27e4cca640b59989489183324255 2023-03-22 17:54:22,062 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/A of 44f11d39aca56af66573ce1ce740420b into a6fc27e4cca640b59989489183324255(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:22,063 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:22,063 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/A, priority=13, startTime=1679507661918; duration=0sec 2023-03-22 17:54:22,063 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:22,063 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:22,064 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:22,064 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/C is initiating minor compaction (all files) 2023-03-22 17:54:22,064 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/C in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:22,064 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/122e97675054449ab56e466b1c21bda3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/4572c1fd3184423d84e0891d7b10cfa1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6cbb8c299a76404bbe2e5b3c457d5f16] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=35.3 K 2023-03-22 17:54:22,065 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322efd75ade186e4e3a8557eb6fddf48fd8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322efd75ade186e4e3a8557eb6fddf48fd8 2023-03-22 17:54:22,065 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 122e97675054449ab56e466b1c21bda3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507659137 2023-03-22 17:54:22,065 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 4572c1fd3184423d84e0891d7b10cfa1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679507659197 2023-03-22 17:54:22,066 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 6cbb8c299a76404bbe2e5b3c457d5f16, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679507661368 2023-03-22 17:54:22,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:22,071 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:22,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507722073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507722074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,081 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#C#compaction#141 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:22,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507722077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507722077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507722078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,102 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/83830810c83040d381c7bdb76b69f75f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83830810c83040d381c7bdb76b69f75f 2023-03-22 17:54:22,106 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/C of 44f11d39aca56af66573ce1ce740420b into 83830810c83040d381c7bdb76b69f75f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:22,107 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:22,107 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/C, priority=13, startTime=1679507661918; duration=0sec 2023-03-22 17:54:22,107 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:22,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507722179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507722179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507722182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507722182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507722183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507722384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507722385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507722386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507722386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507722387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,401 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/0dfe50092fc741aa8c38ae298ea878ff as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/0dfe50092fc741aa8c38ae298ea878ff 2023-03-22 17:54:22,407 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/B of 44f11d39aca56af66573ce1ce740420b into 0dfe50092fc741aa8c38ae298ea878ff(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:22,407 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:22,407 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/B, priority=13, startTime=1679507661918; duration=0sec 2023-03-22 17:54:22,407 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:22,490 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=180, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/b6469c287a90469e8ddd2796f77ab0f2 2023-03-22 17:54:22,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507722689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507722689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507722690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507722690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:22,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507722691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:22,909 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/d54656f82eb34ba98a25911946ffbfb0 2023-03-22 17:54:23,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:23,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507723193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:23,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507723194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:23,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507723195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:23,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507723195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:23,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507723196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,326 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/65e3895b7b0641e9b7997a331ac8e88d 2023-03-22 17:54:23,330 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/b6469c287a90469e8ddd2796f77ab0f2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/b6469c287a90469e8ddd2796f77ab0f2 2023-03-22 17:54:23,335 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/b6469c287a90469e8ddd2796f77ab0f2, entries=150, sequenceid=180, filesize=25.3 K 2023-03-22 17:54:23,335 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/d54656f82eb34ba98a25911946ffbfb0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d54656f82eb34ba98a25911946ffbfb0 2023-03-22 17:54:23,341 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d54656f82eb34ba98a25911946ffbfb0, entries=150, sequenceid=180, filesize=11.8 K 2023-03-22 17:54:23,342 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/65e3895b7b0641e9b7997a331ac8e88d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/65e3895b7b0641e9b7997a331ac8e88d 2023-03-22 17:54:23,346 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/65e3895b7b0641e9b7997a331ac8e88d, entries=150, sequenceid=180, filesize=11.8 K 2023-03-22 17:54:23,347 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=53.67 KB/54960 for 44f11d39aca56af66573ce1ce740420b in 1323ms, sequenceid=180, compaction requested=false 2023-03-22 17:54:23,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:23,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:23,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:23,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:23,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:23,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:23,349 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:23,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:23,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd039fb[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:23,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:23,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:23,352 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:23,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:23,353 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,353 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:23,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:23,355 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:23,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:23,358 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:23,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:23,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:23,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:23,358 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:23,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:23,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:23,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:23,360 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:23,362 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:23,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:23,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:23,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:23,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:23,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:23,366 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:23,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,366 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:23,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:23,466 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:23,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:23,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:23,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:23,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:23,469 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:23,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:23,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:23,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:23,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:23,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:23,471 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:23,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:23,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:23,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:23,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:23,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:23,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:23,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:23,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:23,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:23,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:23,473 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:23,473 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:23,473 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:54:23,473 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:23,474 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:23,474 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:23,474 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:23,474 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:23,474 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:23,887 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322192d03647bef4c83b44b8c4068b8c4db to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322192d03647bef4c83b44b8c4068b8c4db 2023-03-22 17:54:23,894 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=194, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/6530a71e3fae4efcb4f3f6ff3ec6451c 2023-03-22 17:54:24,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:24,198 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:24,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507724212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507724213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507724216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507724218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507664209, max timestamp=1679507664218Num uniques 859; 2023-03-22 17:54:24,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507664209, max timestamp=1679507664218Num uniques 859; 2023-03-22 17:54:24,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507664209, max timestamp=1679507664218Num uniques 859; 2023-03-22 17:54:24,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:54:24,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:54:24,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:54:24,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507724258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,305 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/3b6c3f2a6bde4066af3c4119f734e54e 2023-03-22 17:54:24,316 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/d67ee0104aab46b8aa0cd5d2569ad914 2023-03-22 17:54:24,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/6530a71e3fae4efcb4f3f6ff3ec6451c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6530a71e3fae4efcb4f3f6ff3ec6451c 2023-03-22 17:54:24,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507724321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507724321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507724321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507724321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,331 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6530a71e3fae4efcb4f3f6ff3ec6451c, entries=150, sequenceid=194, filesize=25.3 K 2023-03-22 17:54:24,332 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/3b6c3f2a6bde4066af3c4119f734e54e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3b6c3f2a6bde4066af3c4119f734e54e 2023-03-22 17:54:24,336 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3b6c3f2a6bde4066af3c4119f734e54e, entries=150, sequenceid=194, filesize=11.8 K 2023-03-22 17:54:24,337 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/d67ee0104aab46b8aa0cd5d2569ad914 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d67ee0104aab46b8aa0cd5d2569ad914 2023-03-22 17:54:24,342 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d67ee0104aab46b8aa0cd5d2569ad914, entries=150, sequenceid=194, filesize=11.8 K 2023-03-22 17:54:24,343 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 44f11d39aca56af66573ce1ce740420b in 870ms, sequenceid=194, compaction requested=true 2023-03-22 17:54:24,343 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:24,343 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:24,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2342f768[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:24,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:24,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:24,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:24,352 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:24,354 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:24,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:24,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:24,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:24,354 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:24,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:24,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,357 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,361 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:24,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:24,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,361 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:24,362 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:24,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:24,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:24,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:24,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-22 17:54:24,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:24,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:24,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:24,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:24,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:24,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:24,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507724387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,462 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:24,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:24,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:24,465 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:24,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:24,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:24,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:24,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:24,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:24,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:24,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:24,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:24,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:24,468 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,469 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:24,469 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:24,469 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:24,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:24,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:24,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:24,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c754ea6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:24,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:24,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,474 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,474 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:24,474 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:24,474 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:24,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:24,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:24,478 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:24,478 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:24,478 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:24,478 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:24,478 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:24,480 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,480 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,480 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,481 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,483 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,486 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:24,486 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:24,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:24,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,487 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:24,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:24,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:24,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:24,487 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:24,487 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:24,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507724492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507724524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507724525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507724526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507724527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,587 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:24,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:24,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:24,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:24,596 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:24,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:24,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:24,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:24,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,598 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:24,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:24,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:24,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:24,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:24,616 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,616 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:24,617 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:24,617 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:24,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:24,619 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,619 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:24,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55d13f13[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:24,623 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:24,623 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:24,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,625 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:24,626 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:24,626 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:24,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:24,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:24,628 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:24,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:24,633 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:24,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:24,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:24,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:24,633 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:24,635 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,635 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:24,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,637 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,640 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,650 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:24,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:24,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:24,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:24,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:24,652 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:24,652 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:24,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40418 deadline: 1679507724696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,752 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:24,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:24,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:24,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:24,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:24,780 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:24,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:24,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:24,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:24,781 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,781 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,782 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:24,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:24,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:24,785 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322600505ef53564dce9a0b8982596c2ba5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322600505ef53564dce9a0b8982596c2ba5 2023-03-22 17:54:24,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:24,800 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,800 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:24,800 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:24,800 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:24,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:24,811 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:24,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe4dbb3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:24,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:24,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:24,816 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,816 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:24,817 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:24,822 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:24,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:24,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:24,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:24,822 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:24,825 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,825 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:24,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,825 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:24,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:24,825 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:24,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:24,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:24,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:24,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:24,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,831 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:24,831 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:24,832 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:24,832 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:24,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:24,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:24,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507724827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507724828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507724830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:24,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507724830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,840 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=229, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/1ff24491e76c45359ff4443ee6d416f9 2023-03-22 17:54:24,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/87c1d7acde96455fa6662fb71a57f7cd 2023-03-22 17:54:24,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/83948ca8c8524febad7b7272865e547c 2023-03-22 17:54:24,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/1ff24491e76c45359ff4443ee6d416f9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1ff24491e76c45359ff4443ee6d416f9 2023-03-22 17:54:24,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1ff24491e76c45359ff4443ee6d416f9, entries=200, sequenceid=229, filesize=32.1 K 2023-03-22 17:54:24,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/87c1d7acde96455fa6662fb71a57f7cd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/87c1d7acde96455fa6662fb71a57f7cd 2023-03-22 17:54:24,932 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:24,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:24,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:24,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/87c1d7acde96455fa6662fb71a57f7cd, entries=150, sequenceid=229, filesize=11.8 K 2023-03-22 17:54:24,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/83948ca8c8524febad7b7272865e547c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83948ca8c8524febad7b7272865e547c 2023-03-22 17:54:24,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:24,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:24,950 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:24,950 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:24,950 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:24,950 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:24,950 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,950 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,951 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:24,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:24,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:24,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:24,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:24,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:24,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:24,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:24,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:24,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83948ca8c8524febad7b7272865e547c, entries=150, sequenceid=229, filesize=11.8 K 2023-03-22 17:54:24,955 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:24,955 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:24,955 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 44f11d39aca56af66573ce1ce740420b in 588ms, sequenceid=229, compaction requested=true 2023-03-22 17:54:24,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:24,955 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,955 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:24,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:24,956 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:54:24,956 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:54:24,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:24,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:24,956 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:54:24,956 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:24,957 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:24,957 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:24,957 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:24,957 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110916 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:54:24,957 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/A is initiating minor compaction (all files) 2023-03-22 17:54:24,957 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:24,958 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/A in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,958 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a6fc27e4cca640b59989489183324255, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/b6469c287a90469e8ddd2796f77ab0f2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6530a71e3fae4efcb4f3f6ff3ec6451c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1ff24491e76c45359ff4443ee6d416f9] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=108.3 K 2023-03-22 17:54:24,958 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:54:24,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:24,958 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/B is initiating minor compaction (all files) 2023-03-22 17:54:24,959 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/B in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:24,959 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/0dfe50092fc741aa8c38ae298ea878ff, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d54656f82eb34ba98a25911946ffbfb0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3b6c3f2a6bde4066af3c4119f734e54e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/87c1d7acde96455fa6662fb71a57f7cd] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=47.3 K 2023-03-22 17:54:24,959 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a6fc27e4cca640b59989489183324255, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679507661368 2023-03-22 17:54:24,959 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0dfe50092fc741aa8c38ae298ea878ff, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679507661368 2023-03-22 17:54:24,960 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d54656f82eb34ba98a25911946ffbfb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1679507661433 2023-03-22 17:54:24,960 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 3b6c3f2a6bde4066af3c4119f734e54e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679507662072 2023-03-22 17:54:24,960 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 87c1d7acde96455fa6662fb71a57f7cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679507664256 2023-03-22 17:54:24,964 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b6469c287a90469e8ddd2796f77ab0f2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1679507661433 2023-03-22 17:54:24,965 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6530a71e3fae4efcb4f3f6ff3ec6451c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679507662072 2023-03-22 17:54:24,965 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1ff24491e76c45359ff4443ee6d416f9, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679507664254 2023-03-22 17:54:24,977 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#B#compaction#151 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:25,015 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#A#compaction#150 average throughput is 0.51 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:25,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:25,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:25,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507665045, max timestamp=1679507665074Num uniques 859; 2023-03-22 17:54:25,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:25,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507665045, max timestamp=1679507665074Num uniques 859; 2023-03-22 17:54:25,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507665045, max timestamp=1679507665074Num uniques 859; 2023-03-22 17:54:25,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:25,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:25,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:25,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40418 deadline: 1679507725125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:25,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:25,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40418 deadline: 1679507725226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:25,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507725336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:25,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507725338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:25,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:25,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507725341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:25,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:25,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507725341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:25,425 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303228a5f02c05e8647f89c26777e73616b0a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303228a5f02c05e8647f89c26777e73616b0a 2023-03-22 17:54:25,426 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/70aed078c63248cf818cd116a2efff32 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/70aed078c63248cf818cd116a2efff32 2023-03-22 17:54:25,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40418 deadline: 1679507725430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:25,433 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/B of 44f11d39aca56af66573ce1ce740420b into 70aed078c63248cf818cd116a2efff32(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:25,433 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:25,434 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/B, priority=12, startTime=1679507664956; duration=0sec 2023-03-22 17:54:25,434 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:25,434 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:54:25,435 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:54:25,436 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/C is initiating minor compaction (all files) 2023-03-22 17:54:25,436 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/C in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:25,436 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83830810c83040d381c7bdb76b69f75f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/65e3895b7b0641e9b7997a331ac8e88d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d67ee0104aab46b8aa0cd5d2569ad914, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83948ca8c8524febad7b7272865e547c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=47.3 K 2023-03-22 17:54:25,436 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 83830810c83040d381c7bdb76b69f75f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679507661368 2023-03-22 17:54:25,437 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 65e3895b7b0641e9b7997a331ac8e88d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1679507661433 2023-03-22 17:54:25,437 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d67ee0104aab46b8aa0cd5d2569ad914, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679507662072 2023-03-22 17:54:25,438 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 83948ca8c8524febad7b7272865e547c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679507664256 2023-03-22 17:54:25,440 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/f793111e5dfc4cfdbf7ad5a3cf1d8d7b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f793111e5dfc4cfdbf7ad5a3cf1d8d7b 2023-03-22 17:54:25,445 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/A of 44f11d39aca56af66573ce1ce740420b into f793111e5dfc4cfdbf7ad5a3cf1d8d7b(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:25,445 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:25,445 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/A, priority=12, startTime=1679507664955; duration=0sec 2023-03-22 17:54:25,445 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:25,446 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#C#compaction#153 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:25,458 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/61b9b5fec52f4c60892b3bea1f9a46d4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/61b9b5fec52f4c60892b3bea1f9a46d4 2023-03-22 17:54:25,463 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/C of 44f11d39aca56af66573ce1ce740420b into 61b9b5fec52f4c60892b3bea1f9a46d4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:25,463 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:25,464 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/C, priority=12, startTime=1679507664956; duration=0sec 2023-03-22 17:54:25,464 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:25,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:25,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40418 deadline: 1679507725736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:25,834 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/aa7afc724c124f31a687cb00d71701d1 2023-03-22 17:54:25,844 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/7e09f616ad4342889f1bafdcb1bff5a2 2023-03-22 17:54:26,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40418 deadline: 1679507726239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,262 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/7067433117974edb90157e00cf1ffb95 2023-03-22 17:54:26,267 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/aa7afc724c124f31a687cb00d71701d1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/aa7afc724c124f31a687cb00d71701d1 2023-03-22 17:54:26,270 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/aa7afc724c124f31a687cb00d71701d1, entries=100, sequenceid=239, filesize=18.5 K 2023-03-22 17:54:26,272 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/7e09f616ad4342889f1bafdcb1bff5a2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7e09f616ad4342889f1bafdcb1bff5a2 2023-03-22 17:54:26,276 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7e09f616ad4342889f1bafdcb1bff5a2, entries=100, sequenceid=239, filesize=9.4 K 2023-03-22 17:54:26,277 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/7067433117974edb90157e00cf1ffb95 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7067433117974edb90157e00cf1ffb95 2023-03-22 17:54:26,281 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7067433117974edb90157e00cf1ffb95, entries=100, sequenceid=239, filesize=9.4 K 2023-03-22 17:54:26,282 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 44f11d39aca56af66573ce1ce740420b in 1326ms, sequenceid=239, compaction requested=false 2023-03-22 17:54:26,282 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:26,283 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:26,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:26,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:26,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:26,284 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,284 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:26,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b298a90[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:26,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:26,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,290 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,290 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:26,290 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:26,290 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:26,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:26,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:26,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:26,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:26,295 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:26,295 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:26,295 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:26,295 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:26,296 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:26,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,298 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,299 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:26,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:26,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,301 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,301 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:26,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:26,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,301 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:26,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:26,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-22 17:54:26,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:26,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:26,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:26,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:26,344 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:26,344 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:26,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507726352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40376 deadline: 1679507726352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507726353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507726354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,360 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303223d0c1e87f59e47728aaf8b9f792b1ab0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223d0c1e87f59e47728aaf8b9f792b1ab0 2023-03-22 17:54:26,402 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:26,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:26,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:26,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:26,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:26,406 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:26,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:26,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:26,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,407 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,408 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,408 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,408 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:26,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:26,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:26,416 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:26,416 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,416 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:26,419 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,419 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:26,419 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:26,419 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:26,422 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ddfed2d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:26,426 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:26,426 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,427 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:26,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:26,429 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:26,431 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:26,431 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:26,431 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:26,431 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:26,431 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:26,433 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,433 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,433 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,434 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,438 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,438 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,438 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,438 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:26,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:26,439 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:26,439 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:26,438 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,438 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,438 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507726455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507726456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507726458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,539 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:26,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:26,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:26,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:26,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:26,542 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:26,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:26,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:26,543 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,543 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:26,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:26,545 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:26,545 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,545 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:26,545 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:26,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,551 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:26,553 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:26,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61aa6422[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:26,553 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:26,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:26,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:26,559 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:26,559 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:26,559 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:26,559 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:26,559 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:26,561 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,561 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,561 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,561 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,563 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:26,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,565 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,565 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:26,566 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:26,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507726656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507726659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507726659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,666 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:26,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:26,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:26,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:26,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:26,669 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:26,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:26,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:26,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,672 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,672 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:26,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:26,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:26,674 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,674 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:26,674 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:26,674 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:26,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:26,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:26,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:26,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@384a12e7[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:26,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,683 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:26,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:26,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:26,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:26,685 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:26,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:26,688 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:26,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:26,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:26,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:26,689 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:26,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,700 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,700 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,700 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:26,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:26,700 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,701 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,701 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:26,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,765 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/f36f5ce4611340f093d208aeea76e7fe 2023-03-22 17:54:26,801 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:26,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:26,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:26,805 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:26,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:26,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:26,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,806 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,806 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:26,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:26,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:26,808 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:26,808 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:26,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:26,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:26,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:26,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:26,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:26,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:26,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:26,809 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:26,811 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:26,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fb07274[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:26,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:26,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,816 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:26,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:26,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,817 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:26,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:26,822 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:26,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:26,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:26,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:26,822 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:26,823 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,823 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,823 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,824 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,825 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:26,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:26,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:26,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,828 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:26,828 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:26,928 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:26,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:26,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:26,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:26,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:26,932 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:26,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:26,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:26,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,934 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,934 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:26,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:26,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:26,936 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:26,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:26,938 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,939 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,939 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:26,942 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:26,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc4462a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:26,943 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:26,944 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,947 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:26,947 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,953 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:26,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:26,956 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:26,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:26,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:26,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:26,957 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:26,958 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,958 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,958 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:26,958 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,959 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507726959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:26,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:26,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:26,962 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:26,962 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:26,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507726962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:26,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:26,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:26,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507726963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,961 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:26,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,964 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:26,964 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:26,965 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:26,965 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,965 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:26,965 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:26,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:27,062 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:27,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:27,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:27,065 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:27,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:27,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:27,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,067 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,067 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,067 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:27,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:27,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:27,069 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,069 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:27,069 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:27,069 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:27,071 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,071 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:27,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f7da6c4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:27,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:27,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,075 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:27,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:27,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:27,075 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,075 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:27,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,077 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:27,077 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,077 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:27,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:27,079 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:27,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:27,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:27,080 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:27,080 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:27,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,085 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,088 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,088 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,088 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,088 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:27,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:27,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,088 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,089 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:27,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,089 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/23c2c4f015fd40d8a63a9842596e59a5 2023-03-22 17:54:27,189 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:27,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:27,193 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:27,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:27,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:27,199 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,199 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:27,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:27,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:27,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:27,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:27,202 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:27,204 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,204 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,204 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,204 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:27,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274848e6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:27,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:27,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:27,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:27,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,209 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:27,209 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,209 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:27,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:27,212 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:27,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:27,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:27,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:27,212 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:27,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,214 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,214 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,214 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,218 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:27,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,218 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:27,218 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:27,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40418 deadline: 1679507727243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,318 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:27,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:27,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:27,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:27,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:27,322 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:27,322 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,322 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:27,322 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:27,322 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,323 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:27,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:27,325 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:27,326 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,326 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:27,326 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:27,326 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:27,329 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,329 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,329 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,329 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:27,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bd7bda9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:27,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:27,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,334 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,334 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:27,334 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:27,334 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:27,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:27,338 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:27,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:27,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:27,342 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:27,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:27,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:27,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:27,342 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:27,345 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,345 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,345 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,347 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:27,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:27,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,351 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:27,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,352 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:27,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,452 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:27,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:27,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:27,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:27,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:27,455 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:27,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:27,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:27,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,456 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,456 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,457 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,457 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:27,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:27,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:27,458 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:27,458 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,458 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:27,458 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:27,458 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:27,458 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,458 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:27,458 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:27,458 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:27,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:27,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:27,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:27,460 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:27,462 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:27,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44fd4def[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:27,462 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:27,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:27,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:27,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:27,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507727462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:27,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507727465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:27,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507727465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:27,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:27,470 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:27,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:27,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:27,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:27,470 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:27,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:27,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,475 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:27,475 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:27,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,575 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:27,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:27,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:27,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:27,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:27,578 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:27,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:27,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:27,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,582 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:27,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:27,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:27,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:27,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:27,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:27,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:27,583 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,584 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:27,584 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:27,584 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:27,586 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,586 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,587 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,587 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66948832[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:27,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:27,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,589 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:27,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:27,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:27,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,590 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:27,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:27,593 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:27,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:27,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:27,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:27,594 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:27,596 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,596 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,596 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:27,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:27,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:27,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:27,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:27,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:27,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:27,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:27,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,602 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:27,602 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:27,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:27,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/e1aac8dec0ae4bf180bd67b96e4fb223 2023-03-22 17:54:27,609 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/f36f5ce4611340f093d208aeea76e7fe as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f36f5ce4611340f093d208aeea76e7fe 2023-03-22 17:54:27,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f36f5ce4611340f093d208aeea76e7fe, entries=200, sequenceid=279, filesize=32.3 K 2023-03-22 17:54:27,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/23c2c4f015fd40d8a63a9842596e59a5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/23c2c4f015fd40d8a63a9842596e59a5 2023-03-22 17:54:27,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/23c2c4f015fd40d8a63a9842596e59a5, entries=150, sequenceid=279, filesize=11.9 K 2023-03-22 17:54:27,618 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/e1aac8dec0ae4bf180bd67b96e4fb223 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e1aac8dec0ae4bf180bd67b96e4fb223 2023-03-22 17:54:27,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e1aac8dec0ae4bf180bd67b96e4fb223, entries=150, sequenceid=279, filesize=11.9 K 2023-03-22 17:54:27,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 44f11d39aca56af66573ce1ce740420b in 1285ms, sequenceid=279, compaction requested=true 2023-03-22 17:54:27,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:27,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:27,629 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:27,629 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:27,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:27,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:27,630 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78358 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:27,630 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/A is initiating minor compaction (all files) 2023-03-22 17:54:27,630 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/A in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,630 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f793111e5dfc4cfdbf7ad5a3cf1d8d7b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/aa7afc724c124f31a687cb00d71701d1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f36f5ce4611340f093d208aeea76e7fe] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=76.5 K 2023-03-22 17:54:27,632 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f793111e5dfc4cfdbf7ad5a3cf1d8d7b, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679507664256 2023-03-22 17:54:27,632 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:27,632 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/B is initiating minor compaction (all files) 2023-03-22 17:54:27,632 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/B in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,632 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/70aed078c63248cf818cd116a2efff32, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7e09f616ad4342889f1bafdcb1bff5a2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/23c2c4f015fd40d8a63a9842596e59a5] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=33.5 K 2023-03-22 17:54:27,633 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting aa7afc724c124f31a687cb00d71701d1, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679507664383 2023-03-22 17:54:27,633 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 70aed078c63248cf818cd116a2efff32, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679507664256 2023-03-22 17:54:27,633 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f36f5ce4611340f093d208aeea76e7fe, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679507665106 2023-03-22 17:54:27,634 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7e09f616ad4342889f1bafdcb1bff5a2, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679507664383 2023-03-22 17:54:27,635 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 23c2c4f015fd40d8a63a9842596e59a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679507665116 2023-03-22 17:54:27,644 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#B#compaction#160 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:27,647 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#A#compaction#159 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:27,664 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/5383e0bdae2f49b3a7c56e5171e85ab0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5383e0bdae2f49b3a7c56e5171e85ab0 2023-03-22 17:54:27,669 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/B of 44f11d39aca56af66573ce1ce740420b into 5383e0bdae2f49b3a7c56e5171e85ab0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:27,669 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:27,669 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/B, priority=13, startTime=1679507667629; duration=0sec 2023-03-22 17:54:27,669 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:27,669 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:27,674 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/d1cbc0a0a3f84001be999e34b5f4b846 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d1cbc0a0a3f84001be999e34b5f4b846 2023-03-22 17:54:27,674 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:27,674 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/C is initiating minor compaction (all files) 2023-03-22 17:54:27,674 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/C in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,674 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/61b9b5fec52f4c60892b3bea1f9a46d4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7067433117974edb90157e00cf1ffb95, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e1aac8dec0ae4bf180bd67b96e4fb223] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=33.5 K 2023-03-22 17:54:27,675 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 61b9b5fec52f4c60892b3bea1f9a46d4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679507664256 2023-03-22 17:54:27,676 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7067433117974edb90157e00cf1ffb95, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679507664383 2023-03-22 17:54:27,677 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e1aac8dec0ae4bf180bd67b96e4fb223, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679507665116 2023-03-22 17:54:27,679 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/A of 44f11d39aca56af66573ce1ce740420b into d1cbc0a0a3f84001be999e34b5f4b846(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:27,680 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:27,680 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/A, priority=13, startTime=1679507667629; duration=0sec 2023-03-22 17:54:27,680 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:27,684 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:27,702 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:27,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:27,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:27,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:27,707 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:27,707 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,707 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:27,707 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:27,708 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,708 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,709 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:27,709 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:27,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:27,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:27,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:27,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:27,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:27,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:27,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:27,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:27,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:27,712 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:27,712 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:27,712 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-22 17:54:27,712 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:27,712 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:27,713 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:27,713 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:27,713 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:27,713 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:28,096 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/68645b37324945b6a6f54f65c1b00a48 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/68645b37324945b6a6f54f65c1b00a48 2023-03-22 17:54:28,101 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/C of 44f11d39aca56af66573ce1ce740420b into 68645b37324945b6a6f54f65c1b00a48(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:28,101 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:28,101 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/C, priority=13, startTime=1679507667629; duration=0sec 2023-03-22 17:54:28,101 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:28,138 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032294e2dd1330284548914408d3769f1660 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032294e2dd1330284548914408d3769f1660 2023-03-22 17:54:28,145 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/99c842ef371a47329a3d60cd9287b47f 2023-03-22 17:54:28,188 WARN [DataStreamer for file /user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/fd66ff95aa824aa8b19a5aae6f60663c] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-22 17:54:28,188 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/fd66ff95aa824aa8b19a5aae6f60663c 2023-03-22 17:54:28,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:28,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:28,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507668395, max timestamp=1679507668408Num uniques 859; 2023-03-22 17:54:28,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:54:28,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507668395, max timestamp=1679507668408Num uniques 859; 2023-03-22 17:54:28,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:54:28,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507668395, max timestamp=1679507668408Num uniques 859; 2023-03-22 17:54:28,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:54:28,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507728472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679507668424, max timestamp=1679507668473Num uniques 1018; 2023-03-22 17:54:28,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679507668424, max timestamp=1679507668473Num uniques 1018; 2023-03-22 17:54:28,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679507668424, max timestamp=1679507668473Num uniques 1018; 2023-03-22 17:54:28,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-22 17:54:28,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-22 17:54:28,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-22 17:54:28,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507728498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507728499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507728500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40400 deadline: 1679507728577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507728600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507728601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507728601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,625 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/e21ad0c44c41469186980ccd637ab882 2023-03-22 17:54:28,631 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/99c842ef371a47329a3d60cd9287b47f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/99c842ef371a47329a3d60cd9287b47f 2023-03-22 17:54:28,648 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/99c842ef371a47329a3d60cd9287b47f, entries=150, sequenceid=290, filesize=25.4 K 2023-03-22 17:54:28,648 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/fd66ff95aa824aa8b19a5aae6f60663c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fd66ff95aa824aa8b19a5aae6f60663c 2023-03-22 17:54:28,655 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fd66ff95aa824aa8b19a5aae6f60663c, entries=150, sequenceid=290, filesize=11.9 K 2023-03-22 17:54:28,656 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/e21ad0c44c41469186980ccd637ab882 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e21ad0c44c41469186980ccd637ab882 2023-03-22 17:54:28,663 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e21ad0c44c41469186980ccd637ab882, entries=150, sequenceid=290, filesize=11.9 K 2023-03-22 17:54:28,664 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 44f11d39aca56af66573ce1ce740420b in 952ms, sequenceid=290, compaction requested=false 2023-03-22 17:54:28,664 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:28,664 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:28,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:28,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:28,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:28,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:28,667 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:28,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fef37e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:28,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:28,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,671 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,672 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:28,672 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:28,672 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:28,672 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,672 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:28,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:28,677 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:28,677 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:28,677 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:28,677 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:28,677 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:28,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:28,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:28,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:28,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:28,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,693 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:28,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:28,693 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:28,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:28,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=154.31 KB heapSize=429.84 KB 2023-03-22 17:54:28,784 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:28,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:28,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:28,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:28,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:28,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:28,794 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:28,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:28,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:28,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:28,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:28,797 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:28,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:28,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:28,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:28,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,799 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:28,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:28,799 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,800 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,800 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:28,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:28,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:28,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:28,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:28,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:28,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507728800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507728803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40410 deadline: 1679507728803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40380 deadline: 1679507728803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,809 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322abf02a6803d843769bd9c817c1d99de3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322abf02a6803d843769bd9c817c1d99de3 2023-03-22 17:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:28,812 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:28,812 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:28,812 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:28,812 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:28,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:28,814 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e23c5d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:28,816 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:28,816 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,817 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:28,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:28,819 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:28,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,820 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,820 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:28,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:28,822 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:28,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:28,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:28,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:28,822 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:28,824 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,824 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,824 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:28,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,824 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,826 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:28,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:28,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:28,828 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:28,828 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:28,832 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:28,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:28,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:28,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507728906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,928 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:28,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:28,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:28,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:28,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:28,931 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:28,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:28,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:28,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:28,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,933 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:28,933 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:28,934 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,934 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:28,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:28,942 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:28,942 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:28,942 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:28,942 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:28,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:28,945 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bde9f3b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:28,947 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:28,947 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,948 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:28,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,949 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:28,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:28,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:28,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:28,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:28,952 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:28,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:28,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:28,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:28,952 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:28,954 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,954 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:28,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:28,956 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:28,958 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:28,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:28,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:28,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:28,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:28,961 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:28,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:28,962 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:28,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:28,962 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:28,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:28,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:28,962 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:28,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,062 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:29,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:29,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:29,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:29,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:29,074 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:29,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:29,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:29,075 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,075 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:29,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:29,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:29,078 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,078 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:29,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:29,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:29,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:29,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:29,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:29,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:29,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:29,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:29,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,089 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:29,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c9436a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:29,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:29,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,095 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:29,095 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:29,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:29,095 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:29,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:29,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,097 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:29,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:29,102 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:29,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:29,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:29,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:29,103 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:29,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,105 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507729109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507729110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507729110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507729111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:29,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,115 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,115 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,115 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:29,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,215 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:29,219 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=345, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/6d2e7a5a53d047918378fe7ca895fd5c 2023-03-22 17:54:29,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:29,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:29,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:29,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:29,228 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:29,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:29,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:29,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,230 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,230 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:29,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:29,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:29,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:29,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,236 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:29,240 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:29,240 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:29,240 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,240 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:29,240 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:29,240 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,240 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,244 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/b7d47d42a88b4534b6fa444e2f43f06d 2023-03-22 17:54:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40418 deadline: 1679507729246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:29,249 DEBUG [Thread-864] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4122 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:29,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:29,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d582c2e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:29,251 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:29,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:29,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:29,257 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:29,257 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:29,257 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:29,257 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:29,257 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:29,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,264 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,266 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,266 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,266 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,266 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,267 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:29,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:29,266 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,267 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,267 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,267 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,267 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,267 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:29,268 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:29,368 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:29,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:29,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:29,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:29,371 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:29,371 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,371 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:29,371 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:29,372 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,372 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,373 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,373 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,373 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,374 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:29,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:29,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:29,375 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,375 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:29,375 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:29,375 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:29,377 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,378 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,378 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:29,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96cef0c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:29,379 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:29,379 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,381 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:29,381 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,381 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:29,383 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:29,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:29,385 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:29,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:29,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:29,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:29,385 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:29,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,387 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,387 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,391 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:29,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,391 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:29,391 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:29,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:29,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507729414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,491 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:29,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:29,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:29,495 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:29,495 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,496 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:29,496 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:29,496 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,496 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,498 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:29,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:29,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:29,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:29,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:29,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:29,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:29,502 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:29,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:29,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:29,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,505 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,505 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:29,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76349e5d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:29,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:29,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:29,508 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,508 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:29,509 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:29,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:29,512 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:29,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:29,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:29,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:29,512 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:29,514 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,514 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,514 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,516 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:29,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:29,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:29,521 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:29,521 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:29,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507729612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507729613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:29,621 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:29,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507729616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:29,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:29,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:29,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:29,625 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:29,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:29,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:29,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:29,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:29,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:29,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:29,630 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,630 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:29,631 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:29,631 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:29,631 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:29,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:29,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75e541e2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:29,643 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:29,643 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:29,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:29,646 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:29,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:29,648 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:29,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:29,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:29,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:29,648 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:29,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:29,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,650 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,651 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,654 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:29,654 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,654 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:29,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:29,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:29,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:29,655 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:29,655 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:29,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:29,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:29,686 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/1073c88280b1486694782a51906d578f 2023-03-22 17:54:29,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/6d2e7a5a53d047918378fe7ca895fd5c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6d2e7a5a53d047918378fe7ca895fd5c 2023-03-22 17:54:29,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6d2e7a5a53d047918378fe7ca895fd5c, entries=200, sequenceid=345, filesize=32.3 K 2023-03-22 17:54:29,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/b7d47d42a88b4534b6fa444e2f43f06d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b7d47d42a88b4534b6fa444e2f43f06d 2023-03-22 17:54:29,700 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b7d47d42a88b4534b6fa444e2f43f06d, entries=150, sequenceid=345, filesize=11.9 K 2023-03-22 17:54:29,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/1073c88280b1486694782a51906d578f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1073c88280b1486694782a51906d578f 2023-03-22 17:54:29,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1073c88280b1486694782a51906d578f, entries=150, sequenceid=345, filesize=11.9 K 2023-03-22 17:54:29,706 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.80 KB/440112, currentSize=33.54 KB/34350 for 44f11d39aca56af66573ce1ce740420b in 921ms, sequenceid=345, compaction requested=true 2023-03-22 17:54:29,706 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:29,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:29,706 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:29,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:29,706 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:29,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:29,707 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:29,707 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/A is initiating minor compaction (all files) 2023-03-22 17:54:29,707 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:29,707 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/A in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,707 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/B is initiating minor compaction (all files) 2023-03-22 17:54:29,707 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d1cbc0a0a3f84001be999e34b5f4b846, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/99c842ef371a47329a3d60cd9287b47f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6d2e7a5a53d047918378fe7ca895fd5c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=83.7 K 2023-03-22 17:54:29,707 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/B in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,707 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5383e0bdae2f49b3a7c56e5171e85ab0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fd66ff95aa824aa8b19a5aae6f60663c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b7d47d42a88b4534b6fa444e2f43f06d] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=36.3 K 2023-03-22 17:54:29,708 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d1cbc0a0a3f84001be999e34b5f4b846, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679507665116 2023-03-22 17:54:29,708 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 5383e0bdae2f49b3a7c56e5171e85ab0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679507665116 2023-03-22 17:54:29,708 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 99c842ef371a47329a3d60cd9287b47f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679507666349 2023-03-22 17:54:29,712 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting fd66ff95aa824aa8b19a5aae6f60663c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679507666349 2023-03-22 17:54:29,712 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 6d2e7a5a53d047918378fe7ca895fd5c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679507668495 2023-03-22 17:54:29,713 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b7d47d42a88b4534b6fa444e2f43f06d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679507668497 2023-03-22 17:54:29,737 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#B#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:29,740 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#A#compaction#168 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:29,755 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:29,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:29,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:29,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:29,763 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:29,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:29,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:29,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,765 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:29,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:29,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:29,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:29,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:29,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:29,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:29,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:29,767 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,768 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:29,768 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-22 17:54:29,769 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:29,769 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:29,769 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:29,769 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:29,769 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:29,770 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:29,799 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/8e58a10653d647f1b65dd770de8dddbd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/8e58a10653d647f1b65dd770de8dddbd 2023-03-22 17:54:29,804 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/A of 44f11d39aca56af66573ce1ce740420b into 8e58a10653d647f1b65dd770de8dddbd(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:29,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:29,804 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/A, priority=13, startTime=1679507669706; duration=0sec 2023-03-22 17:54:29,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:29,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:29,805 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:29,805 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/C is initiating minor compaction (all files) 2023-03-22 17:54:29,805 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/C in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:29,805 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/68645b37324945b6a6f54f65c1b00a48, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e21ad0c44c41469186980ccd637ab882, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1073c88280b1486694782a51906d578f] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=36.3 K 2023-03-22 17:54:29,806 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 68645b37324945b6a6f54f65c1b00a48, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679507665116 2023-03-22 17:54:29,806 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e21ad0c44c41469186980ccd637ab882, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679507666349 2023-03-22 17:54:29,807 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1073c88280b1486694782a51906d578f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679507668497 2023-03-22 17:54:29,813 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#C#compaction#171 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:29,822 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/9d4e612a492d446baf3bccaa2500566d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/9d4e612a492d446baf3bccaa2500566d 2023-03-22 17:54:29,827 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/C of 44f11d39aca56af66573ce1ce740420b into 9d4e612a492d446baf3bccaa2500566d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:29,828 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:29,828 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/C, priority=13, startTime=1679507669706; duration=0sec 2023-03-22 17:54:29,828 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:29,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:29,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:29,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507669931, max timestamp=1679507669961Num uniques 859; 2023-03-22 17:54:29,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507669931, max timestamp=1679507669961Num uniques 859; 2023-03-22 17:54:29,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:29,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507669931, max timestamp=1679507669961Num uniques 859; 2023-03-22 17:54:29,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:29,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:30,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507669997, max timestamp=1679507670007Num uniques 1068; 2023-03-22 17:54:30,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-22 17:54:30,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507669997, max timestamp=1679507670007Num uniques 1068; 2023-03-22 17:54:30,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-22 17:54:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507669997, max timestamp=1679507670007Num uniques 1068; 2023-03-22 17:54:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-22 17:54:30,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679507670042, max timestamp=1679507670057Num uniques 1027; 2023-03-22 17:54:30,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679507670042, max timestamp=1679507670057Num uniques 1027; 2023-03-22 17:54:30,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679507670042, max timestamp=1679507670057Num uniques 1027; 2023-03-22 17:54:30,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-22 17:54:30,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-22 17:54:30,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-22 17:54:30,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679507670094, max timestamp=1679507670127Num uniques 1086; 2023-03-22 17:54:30,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-22 17:54:30,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679507670094, max timestamp=1679507670127Num uniques 1086; 2023-03-22 17:54:30,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-22 17:54:30,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679507670094, max timestamp=1679507670127Num uniques 1086; 2023-03-22 17:54:30,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-22 17:54:30,171 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/73152d7904ec44f396e13422f7040a95 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/73152d7904ec44f396e13422f7040a95 2023-03-22 17:54:30,177 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/B of 44f11d39aca56af66573ce1ce740420b into 73152d7904ec44f396e13422f7040a95(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:30,177 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:30,177 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/B, priority=13, startTime=1679507669706; duration=0sec 2023-03-22 17:54:30,177 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:30,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679507670162, max timestamp=1679507670186Num uniques 1045; 2023-03-22 17:54:30,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-22 17:54:30,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679507670162, max timestamp=1679507670186Num uniques 1045; 2023-03-22 17:54:30,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-22 17:54:30,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679507670162, max timestamp=1679507670186Num uniques 1045; 2023-03-22 17:54:30,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-22 17:54:30,203 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032216b1ea6858064cf18afd0fd9ed790ada to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032216b1ea6858064cf18afd0fd9ed790ada 2023-03-22 17:54:30,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679507670236, max timestamp=1679507670245Num uniques 1054; 2023-03-22 17:54:30,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-22 17:54:30,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679507670236, max timestamp=1679507670245Num uniques 1054; 2023-03-22 17:54:30,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679507670236, max timestamp=1679507670245Num uniques 1054; 2023-03-22 17:54:30,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-22 17:54:30,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-22 17:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679507670270, max timestamp=1679507670296Num uniques 1013; 2023-03-22 17:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-22 17:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679507670270, max timestamp=1679507670296Num uniques 1013; 2023-03-22 17:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679507670270, max timestamp=1679507670296Num uniques 1013; 2023-03-22 17:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-22 17:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-22 17:54:30,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679507670315, max timestamp=1679507670348Num uniques 1022; 2023-03-22 17:54:30,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-22 17:54:30,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679507670315, max timestamp=1679507670348Num uniques 1022; 2023-03-22 17:54:30,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-22 17:54:30,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679507670315, max timestamp=1679507670348Num uniques 1022; 2023-03-22 17:54:30,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-22 17:54:30,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679507670371, max timestamp=1679507670394Num uniques 1031; 2023-03-22 17:54:30,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-22 17:54:30,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679507670371, max timestamp=1679507670394Num uniques 1031; 2023-03-22 17:54:30,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-22 17:54:30,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679507670371, max timestamp=1679507670394Num uniques 1031; 2023-03-22 17:54:30,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-22 17:54:30,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679507670426, max timestamp=1679507670448Num uniques 1090; 2023-03-22 17:54:30,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-22 17:54:30,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679507670426, max timestamp=1679507670448Num uniques 1090; 2023-03-22 17:54:30,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-22 17:54:30,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679507670426, max timestamp=1679507670448Num uniques 1090; 2023-03-22 17:54:30,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-22 17:54:30,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679507670483, max timestamp=1679507670496Num uniques 1049; 2023-03-22 17:54:30,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-22 17:54:30,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679507670483, max timestamp=1679507670496Num uniques 1049; 2023-03-22 17:54:30,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679507670483, max timestamp=1679507670496Num uniques 1049; 2023-03-22 17:54:30,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-22 17:54:30,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-22 17:54:30,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679507670531, max timestamp=1679507670541Num uniques 1058; 2023-03-22 17:54:30,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-22 17:54:30,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679507670531, max timestamp=1679507670541Num uniques 1058; 2023-03-22 17:54:30,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-22 17:54:30,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679507670531, max timestamp=1679507670541Num uniques 1058; 2023-03-22 17:54:30,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-22 17:54:30,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679507670575, max timestamp=1679507670582Num uniques 1067; 2023-03-22 17:54:30,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-22 17:54:30,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679507670575, max timestamp=1679507670582Num uniques 1067; 2023-03-22 17:54:30,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679507670575, max timestamp=1679507670582Num uniques 1067; 2023-03-22 17:54:30,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-22 17:54:30,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-22 17:54:30,614 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=353, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/2232960dc5d04110bd4e40c563d78ad2 2023-03-22 17:54:30,637 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/b274877ed0264391b77fca8a144fb99e 2023-03-22 17:54:30,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1679507670623, max timestamp=1679507670644Num uniques 1026; 2023-03-22 17:54:30,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=14926, suffix heap size=134672,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=326 2023-03-22 17:54:30,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679507670636, max timestamp=1679507670644Num uniques 1026; 2023-03-22 17:54:30,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-22 17:54:30,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679507670636, max timestamp=1679507670644Num uniques 1026; 2023-03-22 17:54:30,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-22 17:54:30,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507730667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507730669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=385, cellSize=17630, totalHeapSize=40280, min timestamp=1679507670658, max timestamp=1679507670669Num uniques 1035; 2023-03-22 17:54:30,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=385, cellSize=17630, totalHeapSize=40280, min timestamp=1679507670658, max timestamp=1679507670669Num uniques 1035; 2023-03-22 17:54:30,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=17630, suffix heap size=135608,new segment heap size=40280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=385 2023-03-22 17:54:30,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=385, cellSize=17630, totalHeapSize=40280, min timestamp=1679507670658, max timestamp=1679507670669Num uniques 1185; 2023-03-22 17:54:30,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=17630, suffix heap size=151208,new segment heap size=40280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=385 2023-03-22 17:54:30,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=17630, suffix heap size=135608,new segment heap size=40280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=385 2023-03-22 17:54:30,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507730713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507730714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507730773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507730773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507730816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507730819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507730979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:30,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:30,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507730979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507731019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507731021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,067 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/bb02611dd2b0443d95d63ee93eeede1f 2023-03-22 17:54:31,072 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/2232960dc5d04110bd4e40c563d78ad2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/2232960dc5d04110bd4e40c563d78ad2 2023-03-22 17:54:31,077 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/2232960dc5d04110bd4e40c563d78ad2, entries=150, sequenceid=353, filesize=25.4 K 2023-03-22 17:54:31,078 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/b274877ed0264391b77fca8a144fb99e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b274877ed0264391b77fca8a144fb99e 2023-03-22 17:54:31,082 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b274877ed0264391b77fca8a144fb99e, entries=150, sequenceid=353, filesize=11.9 K 2023-03-22 17:54:31,083 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/bb02611dd2b0443d95d63ee93eeede1f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/bb02611dd2b0443d95d63ee93eeede1f 2023-03-22 17:54:31,086 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/bb02611dd2b0443d95d63ee93eeede1f, entries=150, sequenceid=353, filesize=11.9 K 2023-03-22 17:54:31,087 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 44f11d39aca56af66573ce1ce740420b in 1319ms, sequenceid=353, compaction requested=false 2023-03-22 17:54:31,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:31,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:31,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a03a2c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:31,091 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:31,091 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:31,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:31,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:31,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:31,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:31,096 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:31,096 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:31,096 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:31,096 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:31,096 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:31,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:31,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:31,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:31,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:31,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:31,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:31,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:31,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:31,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:31,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:31,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:31,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:31,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:31,107 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:31,107 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:31,107 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:31,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:31,207 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:31,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:31,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:31,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:31,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:31,210 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:31,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:31,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:31,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:31,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:31,212 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:31,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:31,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:31,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:31,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:31,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:31,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:31,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:31,213 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:31,213 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:31,213 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:31,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:31,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:31,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:31,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:31,215 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=154.31 KB heapSize=423.56 KB 2023-03-22 17:54:31,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:31,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:31,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:31,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:31,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:31,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:31,241 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322bbb6554d93d444fb9fda85bb24ab9fea to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322bbb6554d93d444fb9fda85bb24ab9fea 2023-03-22 17:54:31,259 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=632, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/eb4825a8a44a40e9a0f116a2d47fbf8b 2023-03-22 17:54:31,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:31,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:31,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507731291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507731292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507731322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507731325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507731393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507731397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507731595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507731601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,670 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/d0b47e76b3f1489f93429951c07b0cd9 2023-03-22 17:54:31,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507731826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507731832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507731898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:31,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:31,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507731905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,090 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/d5f93644d43649f3af5ac984f5417fb7 2023-03-22 17:54:32,097 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/eb4825a8a44a40e9a0f116a2d47fbf8b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/eb4825a8a44a40e9a0f116a2d47fbf8b 2023-03-22 17:54:32,101 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/eb4825a8a44a40e9a0f116a2d47fbf8b, entries=150, sequenceid=632, filesize=25.4 K 2023-03-22 17:54:32,102 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/d0b47e76b3f1489f93429951c07b0cd9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d0b47e76b3f1489f93429951c07b0cd9 2023-03-22 17:54:32,107 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d0b47e76b3f1489f93429951c07b0cd9, entries=150, sequenceid=632, filesize=11.9 K 2023-03-22 17:54:32,108 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/d5f93644d43649f3af5ac984f5417fb7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d5f93644d43649f3af5ac984f5417fb7 2023-03-22 17:54:32,112 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d5f93644d43649f3af5ac984f5417fb7, entries=150, sequenceid=632, filesize=11.9 K 2023-03-22 17:54:32,113 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~423.52 KB/433680, currentSize=33.54 KB/34350 for 44f11d39aca56af66573ce1ce740420b in 898ms, sequenceid=632, compaction requested=true 2023-03-22 17:54:32,113 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:32,113 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:32,113 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:32,113 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:32,113 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,113 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:32,113 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:32,115 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:32,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:32,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:32,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:32,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:32,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:32,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:32,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675ab09b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:32,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:32,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:32,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:32,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:32,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:32,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:32,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:32,123 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,123 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:32,125 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:32,129 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:32,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:32,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:32,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:32,129 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:32,134 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,134 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,134 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:32,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,134 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:32,138 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:32,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:32,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:32,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:32,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:32,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:32,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:32,139 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:32,139 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:32,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:32,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:32,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:32,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:32,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:32,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:32,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:32,239 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:32,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:32,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:32,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:32,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:32,243 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:32,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:32,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:32,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:32,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:32,246 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:32,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:32,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:32,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:32,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:32,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:32,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:32,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:32,248 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:32,248 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:32,252 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-22 17:54:32,252 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:32,252 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:32,252 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:32,252 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:32,252 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:32,252 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:32,293 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322ac8647d6331d44819846e5ff26ced3cb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ac8647d6331d44819846e5ff26ced3cb 2023-03-22 17:54:32,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:32,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:32,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507672424, max timestamp=1679507672439Num uniques 859; 2023-03-22 17:54:32,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:32,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507672424, max timestamp=1679507672439Num uniques 859; 2023-03-22 17:54:32,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:32,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507672424, max timestamp=1679507672439Num uniques 859; 2023-03-22 17:54:32,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:32,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:32,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507732470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507672453, max timestamp=1679507672469Num uniques 1068; 2023-03-22 17:54:32,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-22 17:54:32,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507672453, max timestamp=1679507672469Num uniques 1068; 2023-03-22 17:54:32,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-22 17:54:32,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679507672453, max timestamp=1679507672469Num uniques 1068; 2023-03-22 17:54:32,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-22 17:54:32,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:32,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507732514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:32,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507732571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:32,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507732619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,727 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=640, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/d48768a7ea5847a0aa760a9eb36a5f6f 2023-03-22 17:54:32,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:32,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507732773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:32,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507732822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:32,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507732833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:32,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:32,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507732834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:33,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507733078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:33,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507733127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,143 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/cb96676121834764b22344d72bbabe64 2023-03-22 17:54:33,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:33,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40418 deadline: 1679507733282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,286 DEBUG [Thread-864] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:33,558 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/46c20ae9d3f8448196fdf86a25b4f0ba 2023-03-22 17:54:33,563 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/d48768a7ea5847a0aa760a9eb36a5f6f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d48768a7ea5847a0aa760a9eb36a5f6f 2023-03-22 17:54:33,568 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d48768a7ea5847a0aa760a9eb36a5f6f, entries=150, sequenceid=640, filesize=25.4 K 2023-03-22 17:54:33,569 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/cb96676121834764b22344d72bbabe64 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/cb96676121834764b22344d72bbabe64 2023-03-22 17:54:33,573 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/cb96676121834764b22344d72bbabe64, entries=150, sequenceid=640, filesize=11.9 K 2023-03-22 17:54:33,574 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/46c20ae9d3f8448196fdf86a25b4f0ba as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/46c20ae9d3f8448196fdf86a25b4f0ba 2023-03-22 17:54:33,578 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/46c20ae9d3f8448196fdf86a25b4f0ba, entries=150, sequenceid=640, filesize=11.9 K 2023-03-22 17:54:33,579 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 44f11d39aca56af66573ce1ce740420b in 1327ms, sequenceid=640, compaction requested=true 2023-03-22 17:54:33,579 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:33,579 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,581 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:33,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:33,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:33,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ce0c5b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:33,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:33,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507672505, max timestamp=1679507673587Num uniques 1127; 2023-03-22 17:54:33,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-22 17:54:33,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507672505, max timestamp=1679507673587Num uniques 1127; 2023-03-22 17:54:33,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-22 17:54:33,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=111.82 KB heapSize=311.67 KB 2023-03-22 17:54:33,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:33,590 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:33,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:33,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:33,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507672505, max timestamp=1679507673587Num uniques 1127; 2023-03-22 17:54:33,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:33,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:33,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:33,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-22 17:54:33,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:33,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:33,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:33,594 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:33,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:33,599 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:33,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:33,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:33,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:33,599 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:33,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:33,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:33,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:33,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:33,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:33,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:33,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,607 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:33,607 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:33,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:33,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:33,611 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322669cc88c471740e49f846821bba4e4e1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322669cc88c471740e49f846821bba4e4e1 2023-03-22 17:54:33,617 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=695, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/d2ec546e3c414c3aaebcc5744ea8efbf 2023-03-22 17:54:33,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/db9d43ae33f64992980d2b31a1213657 2023-03-22 17:54:33,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679507673607, max timestamp=1679507673648Num uniques 859; 2023-03-22 17:54:33,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679507673607, max timestamp=1679507673648Num uniques 859; 2023-03-22 17:54:33,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-22 17:54:33,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-22 17:54:33,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679507673607, max timestamp=1679507673648Num uniques 859; 2023-03-22 17:54:33,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-22 17:54:33,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:33,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507733685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507673676, max timestamp=1679507673688Num uniques 1218; 2023-03-22 17:54:33,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=7688, suffix heap size=154640,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=168 2023-03-22 17:54:33,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507673676, max timestamp=1679507673688Num uniques 1218; 2023-03-22 17:54:33,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=7688, suffix heap size=154640,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=168 2023-03-22 17:54:33,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507673676, max timestamp=1679507673688Num uniques 1218; 2023-03-22 17:54:33,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=7688, suffix heap size=154640,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=168 2023-03-22 17:54:33,707 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:33,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:33,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:33,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:33,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:33,711 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:33,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:33,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:33,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:33,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,713 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:33,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:33,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:33,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:33,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:33,715 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:33,715 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:33,715 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:33,715 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:33,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:33,718 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,718 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,718 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,718 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:33,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:33,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6afbb151[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:33,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,721 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,721 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:33,723 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:33,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:33,726 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:33,726 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:33,726 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:33,726 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:33,726 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:33,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,728 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,729 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,733 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:33,733 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:33,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:33,733 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:33,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:33,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:33,733 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,734 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,734 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,734 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,734 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:33,734 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:33,734 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,734 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:33,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507673752, max timestamp=1679507673767Num uniques 1027; 2023-03-22 17:54:33,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-22 17:54:33,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507673752, max timestamp=1679507673767Num uniques 1027; 2023-03-22 17:54:33,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-22 17:54:33,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507673752, max timestamp=1679507673767Num uniques 1027; 2023-03-22 17:54:33,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-22 17:54:33,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507673796, max timestamp=1679507673824Num uniques 1036; 2023-03-22 17:54:33,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507673796, max timestamp=1679507673824Num uniques 1036; 2023-03-22 17:54:33,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-22 17:54:33,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-22 17:54:33,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507673796, max timestamp=1679507673824Num uniques 1036; 2023-03-22 17:54:33,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-22 17:54:33,834 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:33,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:33,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:33,840 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:33,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:33,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:33,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:33,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:33,848 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:33,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:33,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:33,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:33,850 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:33,850 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:33,850 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:33,850 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:33,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,852 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,853 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,853 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62028778[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:33,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:33,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,857 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,858 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,858 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:33,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:33,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:33,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:33,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,861 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:33,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:33,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:33,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:33,866 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:33,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:33,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:33,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:33,866 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:33,868 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,868 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,868 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:33,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:33,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:33,869 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:33,870 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:33,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679507673837, max timestamp=1679507673871Num uniques 1095; 2023-03-22 17:54:33,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-22 17:54:33,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679507673837, max timestamp=1679507673871Num uniques 1095; 2023-03-22 17:54:33,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-22 17:54:33,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679507673837, max timestamp=1679507673871Num uniques 1095; 2023-03-22 17:54:33,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-22 17:54:33,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:33,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:33,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:33,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:33,877 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:33,877 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:33,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:33,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:33,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:33,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:33,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:33,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:33,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:33,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679507673896, max timestamp=1679507673906Num uniques 1054; 2023-03-22 17:54:33,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679507673896, max timestamp=1679507673906Num uniques 1054; 2023-03-22 17:54:33,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-22 17:54:33,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-22 17:54:33,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679507673896, max timestamp=1679507673906Num uniques 1054; 2023-03-22 17:54:33,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-22 17:54:33,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679507673927, max timestamp=1679507673950Num uniques 1063; 2023-03-22 17:54:33,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-22 17:54:33,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679507673927, max timestamp=1679507673950Num uniques 1063; 2023-03-22 17:54:33,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-22 17:54:33,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679507673927, max timestamp=1679507673950Num uniques 1063; 2023-03-22 17:54:33,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-22 17:54:33,977 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:33,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:33,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:33,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:33,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:33,992 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:33,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:33,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:33,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:33,993 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,993 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,994 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:33,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:33,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:33,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:33,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:33,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:33,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:33,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:33,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:33,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:33,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:33,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:33,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:33,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:33,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:33,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:33,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:33,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:33,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:33,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:33,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:34,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,001 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:34,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@506e3dcb[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:34,005 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:34,005 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,006 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:34,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:34,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:34,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:34,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679507673975, max timestamp=1679507674005Num uniques 1122; 2023-03-22 17:54:34,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-22 17:54:34,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679507673975, max timestamp=1679507674005Num uniques 1122; 2023-03-22 17:54:34,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-22 17:54:34,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679507673975, max timestamp=1679507674005Num uniques 1122; 2023-03-22 17:54:34,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-22 17:54:34,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:34,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:34,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:34,014 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:34,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:34,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:34,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:34,014 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:34,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:34,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:34,017 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,018 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:34,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:34,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,023 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:34,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:34,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:34,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:34,023 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:34,023 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:34,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1679507674025, max timestamp=1679507674046Num uniques 1131; 2023-03-22 17:54:34,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=15156, suffix heap size=145592,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=331 2023-03-22 17:54:34,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1679507674025, max timestamp=1679507674046Num uniques 1131; 2023-03-22 17:54:34,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=15156, suffix heap size=145592,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=331 2023-03-22 17:54:34,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1679507674025, max timestamp=1679507674046Num uniques 1131; 2023-03-22 17:54:34,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=15156, suffix heap size=145592,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=331 2023-03-22 17:54:34,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/f4ac711952764f9eb5f9b3b07e0dda70 2023-03-22 17:54:34,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/d2ec546e3c414c3aaebcc5744ea8efbf as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d2ec546e3c414c3aaebcc5744ea8efbf 2023-03-22 17:54:34,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d2ec546e3c414c3aaebcc5744ea8efbf, entries=150, sequenceid=695, filesize=25.4 K 2023-03-22 17:54:34,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/db9d43ae33f64992980d2b31a1213657 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/db9d43ae33f64992980d2b31a1213657 2023-03-22 17:54:34,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/db9d43ae33f64992980d2b31a1213657, entries=150, sequenceid=695, filesize=11.9 K 2023-03-22 17:54:34,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/f4ac711952764f9eb5f9b3b07e0dda70 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/f4ac711952764f9eb5f9b3b07e0dda70 2023-03-22 17:54:34,083 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/f4ac711952764f9eb5f9b3b07e0dda70, entries=150, sequenceid=695, filesize=11.9 K 2023-03-22 17:54:34,084 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~64.50 KB/66048, currentSize=127.47 KB/130530 for 44f11d39aca56af66573ce1ce740420b in 494ms, sequenceid=695, compaction requested=true 2023-03-22 17:54:34,084 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:34,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:34,084 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:54:34,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:34,084 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:54:34,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:34,087 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130912 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:54:34,087 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/A is initiating minor compaction (all files) 2023-03-22 17:54:34,087 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:54:34,087 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/A in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:34,087 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/B is initiating minor compaction (all files) 2023-03-22 17:54:34,087 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/B in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:34,087 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/8e58a10653d647f1b65dd770de8dddbd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/2232960dc5d04110bd4e40c563d78ad2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/eb4825a8a44a40e9a0f116a2d47fbf8b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d48768a7ea5847a0aa760a9eb36a5f6f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d2ec546e3c414c3aaebcc5744ea8efbf] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=127.8 K 2023-03-22 17:54:34,087 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/73152d7904ec44f396e13422f7040a95, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b274877ed0264391b77fca8a144fb99e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d0b47e76b3f1489f93429951c07b0cd9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/cb96676121834764b22344d72bbabe64, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/db9d43ae33f64992980d2b31a1213657] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=60.2 K 2023-03-22 17:54:34,088 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 8e58a10653d647f1b65dd770de8dddbd, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679507668497 2023-03-22 17:54:34,088 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 73152d7904ec44f396e13422f7040a95, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679507668497 2023-03-22 17:54:34,089 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b274877ed0264391b77fca8a144fb99e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679507668789 2023-03-22 17:54:34,089 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2232960dc5d04110bd4e40c563d78ad2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679507668789 2023-03-22 17:54:34,089 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d0b47e76b3f1489f93429951c07b0cd9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679507670697 2023-03-22 17:54:34,090 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting cb96676121834764b22344d72bbabe64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679507671285 2023-03-22 17:54:34,090 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting eb4825a8a44a40e9a0f116a2d47fbf8b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679507670697 2023-03-22 17:54:34,090 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting db9d43ae33f64992980d2b31a1213657, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1679507672509 2023-03-22 17:54:34,090 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d48768a7ea5847a0aa760a9eb36a5f6f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679507671285 2023-03-22 17:54:34,091 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d2ec546e3c414c3aaebcc5744ea8efbf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1679507672509 2023-03-22 17:54:34,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:34,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=140.89 KB heapSize=388.78 KB 2023-03-22 17:54:34,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:34,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:34,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:34,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:34,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:34,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:34,106 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#B#compaction#183 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:34,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507734116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,123 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:34,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:34,125 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#A#compaction#184 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:34,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507734119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:34,146 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:34,146 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:34,146 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:34,146 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:34,146 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,146 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,148 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:34,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,149 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,149 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,149 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:34,150 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/694ddead99c04f59b322f5a2dc43917b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/694ddead99c04f59b322f5a2dc43917b 2023-03-22 17:54:34,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:34,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:34,158 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/B of 44f11d39aca56af66573ce1ce740420b into 694ddead99c04f59b322f5a2dc43917b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:34,158 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:34,158 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/B, priority=11, startTime=1679507674084; duration=0sec 2023-03-22 17:54:34,158 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:34,158 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:54:34,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:34,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:34,159 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:34,159 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:34,159 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:34,159 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:34,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:34,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:34,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:34,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:34,161 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,161 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:34,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15875f49[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:34,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:34,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,163 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:34,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:34,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:34,165 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,165 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,165 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:34,166 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:34,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:34,168 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:34,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:34,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:34,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:34,168 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:34,169 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:54:34,169 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/C is initiating minor compaction (all files) 2023-03-22 17:54:34,169 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/C in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:34,169 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/9d4e612a492d446baf3bccaa2500566d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/bb02611dd2b0443d95d63ee93eeede1f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d5f93644d43649f3af5ac984f5417fb7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/46c20ae9d3f8448196fdf86a25b4f0ba, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/f4ac711952764f9eb5f9b3b07e0dda70] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=60.2 K 2023-03-22 17:54:34,170 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,170 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,170 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:34,171 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,171 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9d4e612a492d446baf3bccaa2500566d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1679507668497 2023-03-22 17:54:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,171 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting bb02611dd2b0443d95d63ee93eeede1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679507668789 2023-03-22 17:54:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,172 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d5f93644d43649f3af5ac984f5417fb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1679507670697 2023-03-22 17:54:34,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,172 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322090f9c6d92d04b2b9b561b8139036287 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322090f9c6d92d04b2b9b561b8139036287 2023-03-22 17:54:34,172 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 46c20ae9d3f8448196fdf86a25b4f0ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679507671285 2023-03-22 17:54:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,173 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f4ac711952764f9eb5f9b3b07e0dda70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1679507672509 2023-03-22 17:54:34,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:34,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:34,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:34,176 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:34,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,177 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:34,177 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:34,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:34,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:34,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:34,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:34,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:34,184 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#C#compaction#186 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:34,184 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=867, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/1a805c14e1f4454ca8abc50b3a8b2763 2023-03-22 17:54:34,196 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/e0cb00755eca4977b3ebce0e5c4623b4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e0cb00755eca4977b3ebce0e5c4623b4 2023-03-22 17:54:34,200 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/C of 44f11d39aca56af66573ce1ce740420b into e0cb00755eca4977b3ebce0e5c4623b4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:34,200 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:34,201 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/C, priority=11, startTime=1679507674085; duration=0sec 2023-03-22 17:54:34,201 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:34,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507734223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507734228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,277 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:34,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:34,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:34,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:34,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:34,280 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:34,280 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:34,280 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:34,280 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:34,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,282 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:34,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:34,282 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,282 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:34,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:34,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:34,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:34,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:34,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:34,291 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:34,291 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. started... 2023-03-22 17:54:34,291 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. as already flushing 2023-03-22 17:54:34,291 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:34,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:34,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:34,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,293 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:34,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22c10591[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:34,295 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:34,295 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,297 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:34,297 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:34,297 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:34,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:34,298 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:34,298 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,298 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,298 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:34,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:34,300 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:34,300 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:34,300 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:34,300 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:34,300 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:34,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:34,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:34,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:34,302 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,303 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,305 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,305 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:34,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,306 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:34,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:34,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:34,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:34,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:34,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:34,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:34,306 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:34,306 DEBUG [Thread-872] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:34,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:34,306 DEBUG [Thread-872] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:34,378 DEBUG [Thread-873] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54451f11 to 127.0.0.1:55285 2023-03-22 17:54:34,378 DEBUG [Thread-873] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:34,379 DEBUG [Thread-875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:55285 2023-03-22 17:54:34,379 DEBUG [Thread-875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:34,383 DEBUG [Thread-879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:55285 2023-03-22 17:54:34,383 DEBUG [Thread-879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:34,385 DEBUG [Thread-877] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:55285 2023-03-22 17:54:34,385 DEBUG [Thread-877] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:34,406 DEBUG [Thread-872] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:34,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:34,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507734426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507734431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=867 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/96ec0f8ab3fb4e039b4b201f149b6e7a 2023-03-22 17:54:34,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40380 deadline: 1679507734730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40410 deadline: 1679507734734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40400 deadline: 1679507734841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,843 DEBUG [Thread-862] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:34,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:34,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40376 deadline: 1679507734848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:34,849 DEBUG [Thread-868] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:35,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=867 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/7ab1d6f83dff40e1bef13d6db189811d 2023-03-22 17:54:35,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/1a805c14e1f4454ca8abc50b3a8b2763 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1a805c14e1f4454ca8abc50b3a8b2763 2023-03-22 17:54:35,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1a805c14e1f4454ca8abc50b3a8b2763, entries=150, sequenceid=867, filesize=25.4 K 2023-03-22 17:54:35,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/96ec0f8ab3fb4e039b4b201f149b6e7a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/96ec0f8ab3fb4e039b4b201f149b6e7a 2023-03-22 17:54:35,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/96ec0f8ab3fb4e039b4b201f149b6e7a, entries=150, sequenceid=867, filesize=11.9 K 2023-03-22 17:54:35,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/7ab1d6f83dff40e1bef13d6db189811d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7ab1d6f83dff40e1bef13d6db189811d 2023-03-22 17:54:35,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7ab1d6f83dff40e1bef13d6db189811d, entries=150, sequenceid=867, filesize=11.9 K 2023-03-22 17:54:35,034 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~388.73 KB/398064, currentSize=46.96 KB/48090 for 44f11d39aca56af66573ce1ce740420b in 940ms, sequenceid=867, compaction requested=false 2023-03-22 17:54:35,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:35,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:35,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:54:35,236 DEBUG [Thread-870] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:55285 2023-03-22 17:54:35,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:35,237 DEBUG [Thread-870] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:35,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:35,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:35,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:35,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:35,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:35,239 DEBUG [Thread-866] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:55285 2023-03-22 17:54:35,239 DEBUG [Thread-866] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:35,357 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/5fa82f6999054b2f9b5ea67bbfc6d966 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/5fa82f6999054b2f9b5ea67bbfc6d966 2023-03-22 17:54:35,361 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/A of 44f11d39aca56af66573ce1ce740420b into 5fa82f6999054b2f9b5ea67bbfc6d966(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-22 17:54:35,361 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:35,361 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/A, priority=11, startTime=1679507674084; duration=1sec 2023-03-22 17:54:35,361 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:35,648 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322d67c0e7b9105473296e9518e740d6a79 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322d67c0e7b9105473296e9518e740d6a79 2023-03-22 17:54:36,053 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=880, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/59b645c9bbb243909db455e818f9c129 2023-03-22 17:54:36,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/ccb3ddb76679470ca99e9e4d45d71727 2023-03-22 17:54:36,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/6c90952aecb04ec1b581c73e60cf1380 2023-03-22 17:54:36,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/59b645c9bbb243909db455e818f9c129 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/59b645c9bbb243909db455e818f9c129 2023-03-22 17:54:36,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/59b645c9bbb243909db455e818f9c129, entries=100, sequenceid=880, filesize=18.6 K 2023-03-22 17:54:36,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/ccb3ddb76679470ca99e9e4d45d71727 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/ccb3ddb76679470ca99e9e4d45d71727 2023-03-22 17:54:36,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/ccb3ddb76679470ca99e9e4d45d71727, entries=100, sequenceid=880, filesize=9.5 K 2023-03-22 17:54:36,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/6c90952aecb04ec1b581c73e60cf1380 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6c90952aecb04ec1b581c73e60cf1380 2023-03-22 17:54:36,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6c90952aecb04ec1b581c73e60cf1380, entries=100, sequenceid=880, filesize=9.5 K 2023-03-22 17:54:36,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=6.71 KB/6870 for 44f11d39aca56af66573ce1ce740420b in 1652ms, sequenceid=880, compaction requested=true 2023-03-22 17:54:36,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:36,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:36,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:36,888 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:36,888 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:36,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:36,889 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 71942 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:36,889 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:36,889 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/A is initiating minor compaction (all files) 2023-03-22 17:54:36,889 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/B is initiating minor compaction (all files) 2023-03-22 17:54:36,889 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/A in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:36,889 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/B in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:36,889 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/5fa82f6999054b2f9b5ea67bbfc6d966, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1a805c14e1f4454ca8abc50b3a8b2763, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/59b645c9bbb243909db455e818f9c129] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=70.3 K 2023-03-22 17:54:36,889 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/694ddead99c04f59b322f5a2dc43917b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/96ec0f8ab3fb4e039b4b201f149b6e7a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/ccb3ddb76679470ca99e9e4d45d71727] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=34.1 K 2023-03-22 17:54:36,889 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5fa82f6999054b2f9b5ea67bbfc6d966, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1679507672509 2023-03-22 17:54:36,890 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 694ddead99c04f59b322f5a2dc43917b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1679507672509 2023-03-22 17:54:36,890 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1a805c14e1f4454ca8abc50b3a8b2763, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679507674075 2023-03-22 17:54:36,890 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 96ec0f8ab3fb4e039b4b201f149b6e7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679507674075 2023-03-22 17:54:36,890 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 59b645c9bbb243909db455e818f9c129, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=880, earliestPutTs=1679507674118 2023-03-22 17:54:36,890 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ccb3ddb76679470ca99e9e4d45d71727, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=880, earliestPutTs=1679507674118 2023-03-22 17:54:36,897 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#B#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:36,911 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#A#compaction#193 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:37,314 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/960e88151ed941779568f744d6713a4f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/960e88151ed941779568f744d6713a4f 2023-03-22 17:54:37,319 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/B of 44f11d39aca56af66573ce1ce740420b into 960e88151ed941779568f744d6713a4f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:37,319 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:37,319 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/B, priority=13, startTime=1679507676888; duration=0sec 2023-03-22 17:54:37,319 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:37,319 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:37,320 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:37,320 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 44f11d39aca56af66573ce1ce740420b/C is initiating minor compaction (all files) 2023-03-22 17:54:37,320 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 44f11d39aca56af66573ce1ce740420b/C in TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:37,320 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e0cb00755eca4977b3ebce0e5c4623b4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7ab1d6f83dff40e1bef13d6db189811d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6c90952aecb04ec1b581c73e60cf1380] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp, totalSize=34.1 K 2023-03-22 17:54:37,321 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e0cb00755eca4977b3ebce0e5c4623b4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1679507672509 2023-03-22 17:54:37,321 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7ab1d6f83dff40e1bef13d6db189811d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679507674075 2023-03-22 17:54:37,322 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6c90952aecb04ec1b581c73e60cf1380, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=880, earliestPutTs=1679507674118 2023-03-22 17:54:37,329 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 44f11d39aca56af66573ce1ce740420b#C#compaction#194 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:37,731 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/8aeeb5d3a0a74d62b166585cc9a2ddc0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/8aeeb5d3a0a74d62b166585cc9a2ddc0 2023-03-22 17:54:37,735 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/A of 44f11d39aca56af66573ce1ce740420b into 8aeeb5d3a0a74d62b166585cc9a2ddc0(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:37,735 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:37,735 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/A, priority=13, startTime=1679507676888; duration=0sec 2023-03-22 17:54:37,735 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:37,738 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/117d44e4d6c740f2b82764763ce2b70e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/117d44e4d6c740f2b82764763ce2b70e 2023-03-22 17:54:37,741 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 44f11d39aca56af66573ce1ce740420b/C of 44f11d39aca56af66573ce1ce740420b into 117d44e4d6c740f2b82764763ce2b70e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:37,741 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:37,741 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b., storeName=44f11d39aca56af66573ce1ce740420b/C, priority=13, startTime=1679507676888; duration=0sec 2023-03-22 17:54:37,741 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:38,855 DEBUG [Thread-862] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b3934fb to 127.0.0.1:55285 2023-03-22 17:54:38,855 DEBUG [Thread-862] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:38,882 DEBUG [Thread-868] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69f15cb4 to 127.0.0.1:55285 2023-03-22 17:54:38,882 DEBUG [Thread-868] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:39,479 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-22 17:54:43,374 DEBUG [Thread-864] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aa5eef5 to 127.0.0.1:55285 2023-03-22 17:54:43,374 DEBUG [Thread-864] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:43,374 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-22 17:54:43,374 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 293 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 170 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 5180 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 5101 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 2177 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 6529 rows 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 2175 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 6525 rows 2023-03-22 17:54:43,375 INFO [Listener at localhost/44491] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-22 17:54:43,375 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33c943f0 to 127.0.0.1:55285 2023-03-22 17:54:43,375 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:54:43,379 INFO [Listener at localhost/44491] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-22 17:54:43,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-22 17:54:43,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:43,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-22 17:54:43,385 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507683385"}]},"ts":"1679507683385"} 2023-03-22 17:54:43,386 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-22 17:54:43,388 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-22 17:54:43,389 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, UNASSIGN}] 2023-03-22 17:54:43,390 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, UNASSIGN 2023-03-22 17:54:43,391 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=44f11d39aca56af66573ce1ce740420b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:43,391 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507683391"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507683391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507683391"}]},"ts":"1679507683391"} 2023-03-22 17:54:43,392 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:54:43,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-22 17:54:43,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:43,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 44f11d39aca56af66573ce1ce740420b, disabling compactions & flushes 2023-03-22 17:54:43,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. after waiting 0 ms 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:43,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 44f11d39aca56af66573ce1ce740420b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=A 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=B 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 44f11d39aca56af66573ce1ce740420b, store=C 2023-03-22 17:54:43,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:43,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-22 17:54:43,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322b13dad9517f14feb907088aa8af4f450 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b13dad9517f14feb907088aa8af4f450 2023-03-22 17:54:43,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-22 17:54:44,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=891, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/1d02e943b60145b59beac578241bcda4 2023-03-22 17:54:44,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-22 17:54:44,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=891 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/728691f770b845fc9a25c3a8c2e14bf0 2023-03-22 17:54:45,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=891 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/76ef41ca276042718b1584b508c14f64 2023-03-22 17:54:45,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/A/1d02e943b60145b59beac578241bcda4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1d02e943b60145b59beac578241bcda4 2023-03-22 17:54:45,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1d02e943b60145b59beac578241bcda4, entries=100, sequenceid=891, filesize=18.6 K 2023-03-22 17:54:45,191 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/B/728691f770b845fc9a25c3a8c2e14bf0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/728691f770b845fc9a25c3a8c2e14bf0 2023-03-22 17:54:45,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/728691f770b845fc9a25c3a8c2e14bf0, entries=100, sequenceid=891, filesize=9.5 K 2023-03-22 17:54:45,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/.tmp/C/76ef41ca276042718b1584b508c14f64 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/76ef41ca276042718b1584b508c14f64 2023-03-22 17:54:45,198 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/76ef41ca276042718b1584b508c14f64, entries=100, sequenceid=891, filesize=9.5 K 2023-03-22 17:54:45,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 44f11d39aca56af66573ce1ce740420b in 1655ms, sequenceid=891, compaction requested=false 2023-03-22 17:54:45,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/9b5f89b5d09140fa878a64fbc8f8c322, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f96105752e624eaaa473f0fe0afb1b79, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/cb3ee42ddc8d42d392f56b1ddd088557, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a274a97230dd47b6adcb6ff6d046875f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/487416a1b17546ea922cee1bbacf7c88, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/22bfc8b330564b098ecc81104cee75df, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f536b79cd4924a7a8b1349e971c0c37f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/73e76b0842904b49b2db95c3fd26ee76, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/58c7ced653174608aabe6579b8f01346, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/e8f5e7cec73c4c2c91e3d6fc4ace1412, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a6fc27e4cca640b59989489183324255, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/b6469c287a90469e8ddd2796f77ab0f2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6530a71e3fae4efcb4f3f6ff3ec6451c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1ff24491e76c45359ff4443ee6d416f9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f793111e5dfc4cfdbf7ad5a3cf1d8d7b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/aa7afc724c124f31a687cb00d71701d1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f36f5ce4611340f093d208aeea76e7fe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d1cbc0a0a3f84001be999e34b5f4b846, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/99c842ef371a47329a3d60cd9287b47f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6d2e7a5a53d047918378fe7ca895fd5c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/8e58a10653d647f1b65dd770de8dddbd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/2232960dc5d04110bd4e40c563d78ad2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/eb4825a8a44a40e9a0f116a2d47fbf8b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d48768a7ea5847a0aa760a9eb36a5f6f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/5fa82f6999054b2f9b5ea67bbfc6d966, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d2ec546e3c414c3aaebcc5744ea8efbf, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1a805c14e1f4454ca8abc50b3a8b2763, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/59b645c9bbb243909db455e818f9c129] to archive 2023-03-22 17:54:45,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:54:45,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/9b5f89b5d09140fa878a64fbc8f8c322 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/9b5f89b5d09140fa878a64fbc8f8c322 2023-03-22 17:54:45,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f96105752e624eaaa473f0fe0afb1b79 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f96105752e624eaaa473f0fe0afb1b79 2023-03-22 17:54:45,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/cb3ee42ddc8d42d392f56b1ddd088557 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/cb3ee42ddc8d42d392f56b1ddd088557 2023-03-22 17:54:45,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a274a97230dd47b6adcb6ff6d046875f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a274a97230dd47b6adcb6ff6d046875f 2023-03-22 17:54:45,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/487416a1b17546ea922cee1bbacf7c88 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/487416a1b17546ea922cee1bbacf7c88 2023-03-22 17:54:45,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/22bfc8b330564b098ecc81104cee75df to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/22bfc8b330564b098ecc81104cee75df 2023-03-22 17:54:45,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f536b79cd4924a7a8b1349e971c0c37f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f536b79cd4924a7a8b1349e971c0c37f 2023-03-22 17:54:45,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/73e76b0842904b49b2db95c3fd26ee76 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/73e76b0842904b49b2db95c3fd26ee76 2023-03-22 17:54:45,231 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/58c7ced653174608aabe6579b8f01346 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/58c7ced653174608aabe6579b8f01346 2023-03-22 17:54:45,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/e8f5e7cec73c4c2c91e3d6fc4ace1412 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/e8f5e7cec73c4c2c91e3d6fc4ace1412 2023-03-22 17:54:45,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a6fc27e4cca640b59989489183324255 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/a6fc27e4cca640b59989489183324255 2023-03-22 17:54:45,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/b6469c287a90469e8ddd2796f77ab0f2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/b6469c287a90469e8ddd2796f77ab0f2 2023-03-22 17:54:45,234 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6530a71e3fae4efcb4f3f6ff3ec6451c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6530a71e3fae4efcb4f3f6ff3ec6451c 2023-03-22 17:54:45,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1ff24491e76c45359ff4443ee6d416f9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1ff24491e76c45359ff4443ee6d416f9 2023-03-22 17:54:45,236 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f793111e5dfc4cfdbf7ad5a3cf1d8d7b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f793111e5dfc4cfdbf7ad5a3cf1d8d7b 2023-03-22 17:54:45,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/aa7afc724c124f31a687cb00d71701d1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/aa7afc724c124f31a687cb00d71701d1 2023-03-22 17:54:45,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f36f5ce4611340f093d208aeea76e7fe to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/f36f5ce4611340f093d208aeea76e7fe 2023-03-22 17:54:45,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d1cbc0a0a3f84001be999e34b5f4b846 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d1cbc0a0a3f84001be999e34b5f4b846 2023-03-22 17:54:45,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/99c842ef371a47329a3d60cd9287b47f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/99c842ef371a47329a3d60cd9287b47f 2023-03-22 17:54:45,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6d2e7a5a53d047918378fe7ca895fd5c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/6d2e7a5a53d047918378fe7ca895fd5c 2023-03-22 17:54:45,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/8e58a10653d647f1b65dd770de8dddbd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/8e58a10653d647f1b65dd770de8dddbd 2023-03-22 17:54:45,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/2232960dc5d04110bd4e40c563d78ad2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/2232960dc5d04110bd4e40c563d78ad2 2023-03-22 17:54:45,243 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/eb4825a8a44a40e9a0f116a2d47fbf8b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/eb4825a8a44a40e9a0f116a2d47fbf8b 2023-03-22 17:54:45,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d48768a7ea5847a0aa760a9eb36a5f6f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d48768a7ea5847a0aa760a9eb36a5f6f 2023-03-22 17:54:45,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/5fa82f6999054b2f9b5ea67bbfc6d966 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/5fa82f6999054b2f9b5ea67bbfc6d966 2023-03-22 17:54:45,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d2ec546e3c414c3aaebcc5744ea8efbf to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/d2ec546e3c414c3aaebcc5744ea8efbf 2023-03-22 17:54:45,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1a805c14e1f4454ca8abc50b3a8b2763 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1a805c14e1f4454ca8abc50b3a8b2763 2023-03-22 17:54:45,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/59b645c9bbb243909db455e818f9c129 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/59b645c9bbb243909db455e818f9c129 2023-03-22 17:54:45,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fca4bfb0375349d9a3c9ac686abdbd04, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/bea102982edf44ad82fdbea4ba31fa53, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3e1dcda1b202406d8faf572379dd3f50, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/28eee1e31b224575b4bb4ab117b0ce80, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/e40c4e6ea62243ada670c1e49b9b6377, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b6ff0b3852124325af927db1ff57ef31, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/2273bbd0bb794a2bacf9c3702aaae1c6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/c9a9662fe7b04f5b804f1072590ad0b2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7ce956c0211a40ef87b57c9ed00ae8a7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/0dfe50092fc741aa8c38ae298ea878ff, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5b1ccf8f3b1247f6bdf06d5354a8350c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d54656f82eb34ba98a25911946ffbfb0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3b6c3f2a6bde4066af3c4119f734e54e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/70aed078c63248cf818cd116a2efff32, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/87c1d7acde96455fa6662fb71a57f7cd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7e09f616ad4342889f1bafdcb1bff5a2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5383e0bdae2f49b3a7c56e5171e85ab0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/23c2c4f015fd40d8a63a9842596e59a5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fd66ff95aa824aa8b19a5aae6f60663c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/73152d7904ec44f396e13422f7040a95, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b7d47d42a88b4534b6fa444e2f43f06d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b274877ed0264391b77fca8a144fb99e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d0b47e76b3f1489f93429951c07b0cd9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/cb96676121834764b22344d72bbabe64, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/694ddead99c04f59b322f5a2dc43917b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/db9d43ae33f64992980d2b31a1213657, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/96ec0f8ab3fb4e039b4b201f149b6e7a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/ccb3ddb76679470ca99e9e4d45d71727] to archive 2023-03-22 17:54:45,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:54:45,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fca4bfb0375349d9a3c9ac686abdbd04 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fca4bfb0375349d9a3c9ac686abdbd04 2023-03-22 17:54:45,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/bea102982edf44ad82fdbea4ba31fa53 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/bea102982edf44ad82fdbea4ba31fa53 2023-03-22 17:54:45,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3e1dcda1b202406d8faf572379dd3f50 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3e1dcda1b202406d8faf572379dd3f50 2023-03-22 17:54:45,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/28eee1e31b224575b4bb4ab117b0ce80 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/28eee1e31b224575b4bb4ab117b0ce80 2023-03-22 17:54:45,257 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/e40c4e6ea62243ada670c1e49b9b6377 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/e40c4e6ea62243ada670c1e49b9b6377 2023-03-22 17:54:45,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b6ff0b3852124325af927db1ff57ef31 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b6ff0b3852124325af927db1ff57ef31 2023-03-22 17:54:45,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/2273bbd0bb794a2bacf9c3702aaae1c6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/2273bbd0bb794a2bacf9c3702aaae1c6 2023-03-22 17:54:45,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/c9a9662fe7b04f5b804f1072590ad0b2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/c9a9662fe7b04f5b804f1072590ad0b2 2023-03-22 17:54:45,261 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7ce956c0211a40ef87b57c9ed00ae8a7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7ce956c0211a40ef87b57c9ed00ae8a7 2023-03-22 17:54:45,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/0dfe50092fc741aa8c38ae298ea878ff to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/0dfe50092fc741aa8c38ae298ea878ff 2023-03-22 17:54:45,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5b1ccf8f3b1247f6bdf06d5354a8350c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5b1ccf8f3b1247f6bdf06d5354a8350c 2023-03-22 17:54:45,264 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d54656f82eb34ba98a25911946ffbfb0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d54656f82eb34ba98a25911946ffbfb0 2023-03-22 17:54:45,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3b6c3f2a6bde4066af3c4119f734e54e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/3b6c3f2a6bde4066af3c4119f734e54e 2023-03-22 17:54:45,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/70aed078c63248cf818cd116a2efff32 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/70aed078c63248cf818cd116a2efff32 2023-03-22 17:54:45,267 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/87c1d7acde96455fa6662fb71a57f7cd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/87c1d7acde96455fa6662fb71a57f7cd 2023-03-22 17:54:45,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7e09f616ad4342889f1bafdcb1bff5a2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/7e09f616ad4342889f1bafdcb1bff5a2 2023-03-22 17:54:45,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5383e0bdae2f49b3a7c56e5171e85ab0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/5383e0bdae2f49b3a7c56e5171e85ab0 2023-03-22 17:54:45,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/23c2c4f015fd40d8a63a9842596e59a5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/23c2c4f015fd40d8a63a9842596e59a5 2023-03-22 17:54:45,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fd66ff95aa824aa8b19a5aae6f60663c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/fd66ff95aa824aa8b19a5aae6f60663c 2023-03-22 17:54:45,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/73152d7904ec44f396e13422f7040a95 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/73152d7904ec44f396e13422f7040a95 2023-03-22 17:54:45,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b7d47d42a88b4534b6fa444e2f43f06d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b7d47d42a88b4534b6fa444e2f43f06d 2023-03-22 17:54:45,274 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b274877ed0264391b77fca8a144fb99e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/b274877ed0264391b77fca8a144fb99e 2023-03-22 17:54:45,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d0b47e76b3f1489f93429951c07b0cd9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/d0b47e76b3f1489f93429951c07b0cd9 2023-03-22 17:54:45,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/cb96676121834764b22344d72bbabe64 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/cb96676121834764b22344d72bbabe64 2023-03-22 17:54:45,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/694ddead99c04f59b322f5a2dc43917b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/694ddead99c04f59b322f5a2dc43917b 2023-03-22 17:54:45,278 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/db9d43ae33f64992980d2b31a1213657 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/db9d43ae33f64992980d2b31a1213657 2023-03-22 17:54:45,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/96ec0f8ab3fb4e039b4b201f149b6e7a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/96ec0f8ab3fb4e039b4b201f149b6e7a 2023-03-22 17:54:45,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/ccb3ddb76679470ca99e9e4d45d71727 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/ccb3ddb76679470ca99e9e4d45d71727 2023-03-22 17:54:45,282 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/b1d2ddc5f90a443bb9615a46c76f7525, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6355064a01934b9aa9680bdab9beefc2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/c7b0e5c03925446184ee7ba61f11e75b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1395644ef8d0482ca71fda83c85a57f0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/532248871c5f4145b6b28dc59c2815f3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/44221fe2a5bd4934be7700edf276ef64, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/122e97675054449ab56e466b1c21bda3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/2ab6131ecee74e99adae1479d1f2343e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/4572c1fd3184423d84e0891d7b10cfa1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83830810c83040d381c7bdb76b69f75f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6cbb8c299a76404bbe2e5b3c457d5f16, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/65e3895b7b0641e9b7997a331ac8e88d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d67ee0104aab46b8aa0cd5d2569ad914, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/61b9b5fec52f4c60892b3bea1f9a46d4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83948ca8c8524febad7b7272865e547c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7067433117974edb90157e00cf1ffb95, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/68645b37324945b6a6f54f65c1b00a48, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e1aac8dec0ae4bf180bd67b96e4fb223, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e21ad0c44c41469186980ccd637ab882, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/9d4e612a492d446baf3bccaa2500566d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1073c88280b1486694782a51906d578f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/bb02611dd2b0443d95d63ee93eeede1f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d5f93644d43649f3af5ac984f5417fb7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/46c20ae9d3f8448196fdf86a25b4f0ba, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e0cb00755eca4977b3ebce0e5c4623b4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/f4ac711952764f9eb5f9b3b07e0dda70, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7ab1d6f83dff40e1bef13d6db189811d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6c90952aecb04ec1b581c73e60cf1380] to archive 2023-03-22 17:54:45,283 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:54:45,284 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/b1d2ddc5f90a443bb9615a46c76f7525 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/b1d2ddc5f90a443bb9615a46c76f7525 2023-03-22 17:54:45,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6355064a01934b9aa9680bdab9beefc2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6355064a01934b9aa9680bdab9beefc2 2023-03-22 17:54:45,286 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/c7b0e5c03925446184ee7ba61f11e75b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/c7b0e5c03925446184ee7ba61f11e75b 2023-03-22 17:54:45,287 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1395644ef8d0482ca71fda83c85a57f0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1395644ef8d0482ca71fda83c85a57f0 2023-03-22 17:54:45,288 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/532248871c5f4145b6b28dc59c2815f3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/532248871c5f4145b6b28dc59c2815f3 2023-03-22 17:54:45,290 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/44221fe2a5bd4934be7700edf276ef64 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/44221fe2a5bd4934be7700edf276ef64 2023-03-22 17:54:45,291 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/122e97675054449ab56e466b1c21bda3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/122e97675054449ab56e466b1c21bda3 2023-03-22 17:54:45,292 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/2ab6131ecee74e99adae1479d1f2343e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/2ab6131ecee74e99adae1479d1f2343e 2023-03-22 17:54:45,293 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/4572c1fd3184423d84e0891d7b10cfa1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/4572c1fd3184423d84e0891d7b10cfa1 2023-03-22 17:54:45,294 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83830810c83040d381c7bdb76b69f75f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83830810c83040d381c7bdb76b69f75f 2023-03-22 17:54:45,295 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6cbb8c299a76404bbe2e5b3c457d5f16 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6cbb8c299a76404bbe2e5b3c457d5f16 2023-03-22 17:54:45,295 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/65e3895b7b0641e9b7997a331ac8e88d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/65e3895b7b0641e9b7997a331ac8e88d 2023-03-22 17:54:45,296 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d67ee0104aab46b8aa0cd5d2569ad914 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d67ee0104aab46b8aa0cd5d2569ad914 2023-03-22 17:54:45,297 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/61b9b5fec52f4c60892b3bea1f9a46d4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/61b9b5fec52f4c60892b3bea1f9a46d4 2023-03-22 17:54:45,298 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83948ca8c8524febad7b7272865e547c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/83948ca8c8524febad7b7272865e547c 2023-03-22 17:54:45,299 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7067433117974edb90157e00cf1ffb95 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7067433117974edb90157e00cf1ffb95 2023-03-22 17:54:45,300 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/68645b37324945b6a6f54f65c1b00a48 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/68645b37324945b6a6f54f65c1b00a48 2023-03-22 17:54:45,301 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e1aac8dec0ae4bf180bd67b96e4fb223 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e1aac8dec0ae4bf180bd67b96e4fb223 2023-03-22 17:54:45,302 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e21ad0c44c41469186980ccd637ab882 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e21ad0c44c41469186980ccd637ab882 2023-03-22 17:54:45,303 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/9d4e612a492d446baf3bccaa2500566d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/9d4e612a492d446baf3bccaa2500566d 2023-03-22 17:54:45,305 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1073c88280b1486694782a51906d578f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/1073c88280b1486694782a51906d578f 2023-03-22 17:54:45,306 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/bb02611dd2b0443d95d63ee93eeede1f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/bb02611dd2b0443d95d63ee93eeede1f 2023-03-22 17:54:45,307 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d5f93644d43649f3af5ac984f5417fb7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/d5f93644d43649f3af5ac984f5417fb7 2023-03-22 17:54:45,308 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/46c20ae9d3f8448196fdf86a25b4f0ba to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/46c20ae9d3f8448196fdf86a25b4f0ba 2023-03-22 17:54:45,309 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e0cb00755eca4977b3ebce0e5c4623b4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/e0cb00755eca4977b3ebce0e5c4623b4 2023-03-22 17:54:45,310 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/f4ac711952764f9eb5f9b3b07e0dda70 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/f4ac711952764f9eb5f9b3b07e0dda70 2023-03-22 17:54:45,311 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7ab1d6f83dff40e1bef13d6db189811d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/7ab1d6f83dff40e1bef13d6db189811d 2023-03-22 17:54:45,312 DEBUG [StoreCloser-TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6c90952aecb04ec1b581c73e60cf1380 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/6c90952aecb04ec1b581c73e60cf1380 2023-03-22 17:54:45,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/recovered.edits/894.seqid, newMaxSeqId=894, maxSeqId=4 2023-03-22 17:54:45,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b. 2023-03-22 17:54:45,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 44f11d39aca56af66573ce1ce740420b: 2023-03-22 17:54:45,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:45,321 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=44f11d39aca56af66573ce1ce740420b, regionState=CLOSED 2023-03-22 17:54:45,322 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507685321"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507685321"}]},"ts":"1679507685321"} 2023-03-22 17:54:45,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-22 17:54:45,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 44f11d39aca56af66573ce1ce740420b, server=jenkins-hbase4.apache.org,34403,1679507617723 in 1.9300 sec 2023-03-22 17:54:45,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-22 17:54:45,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=44f11d39aca56af66573ce1ce740420b, UNASSIGN in 1.9350 sec 2023-03-22 17:54:45,325 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507685325"}]},"ts":"1679507685325"} 2023-03-22 17:54:45,326 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-22 17:54:45,328 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-22 17:54:45,329 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9480 sec 2023-03-22 17:54:45,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-22 17:54:45,489 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-22 17:54:45,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-22 17:54:45,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:45,491 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:45,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-22 17:54:45,492 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:45,497 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:45,499 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/recovered.edits] 2023-03-22 17:54:45,501 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1d02e943b60145b59beac578241bcda4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/1d02e943b60145b59beac578241bcda4 2023-03-22 17:54:45,502 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/8aeeb5d3a0a74d62b166585cc9a2ddc0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/A/8aeeb5d3a0a74d62b166585cc9a2ddc0 2023-03-22 17:54:45,504 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/728691f770b845fc9a25c3a8c2e14bf0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/728691f770b845fc9a25c3a8c2e14bf0 2023-03-22 17:54:45,505 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/960e88151ed941779568f744d6713a4f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/B/960e88151ed941779568f744d6713a4f 2023-03-22 17:54:45,507 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/117d44e4d6c740f2b82764763ce2b70e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/117d44e4d6c740f2b82764763ce2b70e 2023-03-22 17:54:45,509 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/76ef41ca276042718b1584b508c14f64 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/C/76ef41ca276042718b1584b508c14f64 2023-03-22 17:54:45,512 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/recovered.edits/894.seqid to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b/recovered.edits/894.seqid 2023-03-22 17:54:45,513 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/44f11d39aca56af66573ce1ce740420b 2023-03-22 17:54:45,513 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:54:45,514 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-22 17:54:45,514 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-22 17:54:45,518 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322090f9c6d92d04b2b9b561b8139036287 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322090f9c6d92d04b2b9b561b8139036287 2023-03-22 17:54:45,519 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032216b1ea6858064cf18afd0fd9ed790ada to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032216b1ea6858064cf18afd0fd9ed790ada 2023-03-22 17:54:45,520 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322192d03647bef4c83b44b8c4068b8c4db to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322192d03647bef4c83b44b8c4068b8c4db 2023-03-22 17:54:45,521 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223d0c1e87f59e47728aaf8b9f792b1ab0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223d0c1e87f59e47728aaf8b9f792b1ab0 2023-03-22 17:54:45,522 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322600505ef53564dce9a0b8982596c2ba5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322600505ef53564dce9a0b8982596c2ba5 2023-03-22 17:54:45,523 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322669cc88c471740e49f846821bba4e4e1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322669cc88c471740e49f846821bba4e4e1 2023-03-22 17:54:45,524 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032288c6bd2ab44343b4a11a451638315687 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032288c6bd2ab44343b4a11a451638315687 2023-03-22 17:54:45,525 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303228a5f02c05e8647f89c26777e73616b0a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303228a5f02c05e8647f89c26777e73616b0a 2023-03-22 17:54:45,526 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322940aa05c92984500a49ff73a27d8c2fd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322940aa05c92984500a49ff73a27d8c2fd 2023-03-22 17:54:45,526 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032294e2dd1330284548914408d3769f1660 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032294e2dd1330284548914408d3769f1660 2023-03-22 17:54:45,527 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303229db2f2cae98345379ae626ed93d9a7ea to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303229db2f2cae98345379ae626ed93d9a7ea 2023-03-22 17:54:45,528 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322aa113818cf104f87828bcf3e308c2e16 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322aa113818cf104f87828bcf3e308c2e16 2023-03-22 17:54:45,529 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322abf02a6803d843769bd9c817c1d99de3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322abf02a6803d843769bd9c817c1d99de3 2023-03-22 17:54:45,530 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ac8647d6331d44819846e5ff26ced3cb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ac8647d6331d44819846e5ff26ced3cb 2023-03-22 17:54:45,531 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b13dad9517f14feb907088aa8af4f450 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b13dad9517f14feb907088aa8af4f450 2023-03-22 17:54:45,531 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322bbb6554d93d444fb9fda85bb24ab9fea to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322bbb6554d93d444fb9fda85bb24ab9fea 2023-03-22 17:54:45,532 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c5d7120d998348c2a8564e8448184751 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c5d7120d998348c2a8564e8448184751 2023-03-22 17:54:45,533 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c86ac56f3b1c421cb20b2ccb9ae89976 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c86ac56f3b1c421cb20b2ccb9ae89976 2023-03-22 17:54:45,534 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c895344d78d349eea800426886a501fe to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c895344d78d349eea800426886a501fe 2023-03-22 17:54:45,535 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322d67c0e7b9105473296e9518e740d6a79 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322d67c0e7b9105473296e9518e740d6a79 2023-03-22 17:54:45,536 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e3608406c0154aac93ec946d743ab9e3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e3608406c0154aac93ec946d743ab9e3 2023-03-22 17:54:45,537 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322efd75ade186e4e3a8557eb6fddf48fd8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322efd75ade186e4e3a8557eb6fddf48fd8 2023-03-22 17:54:45,538 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f543a0f38dfe4633bcf4a3f53ce0a42c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f543a0f38dfe4633bcf4a3f53ce0a42c 2023-03-22 17:54:45,538 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-22 17:54:45,540 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:45,542 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-22 17:54:45,544 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-22 17:54:45,544 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:45,544 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-22 17:54:45,545 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679507685545"}]},"ts":"9223372036854775807"} 2023-03-22 17:54:45,546 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-22 17:54:45,546 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 44f11d39aca56af66573ce1ce740420b, NAME => 'TestAcidGuarantees,,1679507651253.44f11d39aca56af66573ce1ce740420b.', STARTKEY => '', ENDKEY => ''}] 2023-03-22 17:54:45,546 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-22 17:54:45,546 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679507685546"}]},"ts":"9223372036854775807"} 2023-03-22 17:54:45,547 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-22 17:54:45,552 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:45,553 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 63 msec 2023-03-22 17:54:45,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-22 17:54:45,593 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-22 17:54:45,605 INFO [Listener at localhost/44491] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 231), OpenFileDescriptor=536 (was 545), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=319 (was 278) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6013 (was 6164) 2023-03-22 17:54:45,616 INFO [Listener at localhost/44491] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=536, MaxFileDescriptor=60000, SystemLoadAverage=319, ProcessCount=169, AvailableMemoryMB=6013 2023-03-22 17:54:45,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:54:45,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:54:45,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-22 17:54:45,620 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-22 17:54:45,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-22 17:54:45,621 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-22 17:54:45,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-22 17:54:45,622 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:45,623 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d empty. 2023-03-22 17:54:45,623 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:45,623 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:54:45,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-22 17:54:45,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-22 17:54:46,032 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:54:46,034 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a0a59fb2216419102dce85f7adab989d, NAME => 'TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp 2023-03-22 17:54:46,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-22 17:54:46,441 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:54:46,441 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a0a59fb2216419102dce85f7adab989d, disabling compactions & flushes 2023-03-22 17:54:46,441 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:46,441 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:46,441 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. after waiting 0 ms 2023-03-22 17:54:46,441 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:46,441 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:46,441 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:46,443 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-22 17:54:46,444 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507686444"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507686444"}]},"ts":"1679507686444"} 2023-03-22 17:54:46,445 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-22 17:54:46,445 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-22 17:54:46,445 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507686445"}]},"ts":"1679507686445"} 2023-03-22 17:54:46,446 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-22 17:54:46,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a0a59fb2216419102dce85f7adab989d, ASSIGN}] 2023-03-22 17:54:46,451 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a0a59fb2216419102dce85f7adab989d, ASSIGN 2023-03-22 17:54:46,451 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a0a59fb2216419102dce85f7adab989d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=false 2023-03-22 17:54:46,602 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a0a59fb2216419102dce85f7adab989d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:46,603 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507686602"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507686602"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507686602"}]},"ts":"1679507686602"} 2023-03-22 17:54:46,604 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:54:46,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-22 17:54:46,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:46,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a0a59fb2216419102dce85f7adab989d, NAME => 'TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:54:46,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:54:46,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,762 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,764 DEBUG [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A 2023-03-22 17:54:46,764 DEBUG [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A 2023-03-22 17:54:46,764 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:46,764 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a0a59fb2216419102dce85f7adab989d columnFamilyName A 2023-03-22 17:54:46,765 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.HStore(310): Store=a0a59fb2216419102dce85f7adab989d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:46,765 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,766 DEBUG [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B 2023-03-22 17:54:46,766 DEBUG [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B 2023-03-22 17:54:46,766 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:46,766 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a0a59fb2216419102dce85f7adab989d columnFamilyName B 2023-03-22 17:54:46,767 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.HStore(310): Store=a0a59fb2216419102dce85f7adab989d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:46,767 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,769 DEBUG [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C 2023-03-22 17:54:46,769 DEBUG [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C 2023-03-22 17:54:46,769 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:54:46,769 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a0a59fb2216419102dce85f7adab989d columnFamilyName C 2023-03-22 17:54:46,769 INFO [StoreOpener-a0a59fb2216419102dce85f7adab989d-1] regionserver.HStore(310): Store=a0a59fb2216419102dce85f7adab989d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:54:46,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:46,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:54:46,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:46,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:54:46,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a0a59fb2216419102dce85f7adab989d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61778925, jitterRate=-0.07942228019237518}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:54:46,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:46,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., pid=30, masterSystemTime=1679507686755 2023-03-22 17:54:46,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:46,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:46,779 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a0a59fb2216419102dce85f7adab989d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:46,780 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507686779"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507686779"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507686779"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507686779"}]},"ts":"1679507686779"} 2023-03-22 17:54:46,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-22 17:54:46,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 in 177 msec 2023-03-22 17:54:46,785 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-22 17:54:46,785 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a0a59fb2216419102dce85f7adab989d, ASSIGN in 334 msec 2023-03-22 17:54:46,785 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-22 17:54:46,786 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507686785"}]},"ts":"1679507686785"} 2023-03-22 17:54:46,786 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-22 17:54:46,789 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-22 17:54:46,790 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1710 sec 2023-03-22 17:54:47,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-22 17:54:47,726 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-22 17:54:47,726 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x112921f2 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,732 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22b4eae5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,733 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,735 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54122, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,736 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-22 17:54:47,737 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58854, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-22 17:54:47,739 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,742 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,743 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,746 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,746 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,751 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,751 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,756 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,756 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,760 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,760 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,769 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,769 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,777 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,778 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,783 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,783 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,786 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,786 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:54:47,790 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:54:47,794 DEBUG [hconnection-0x144735b9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,794 DEBUG [hconnection-0x3469fe8f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,794 DEBUG [hconnection-0x492b5c1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,795 DEBUG [hconnection-0x5ae59228-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,795 DEBUG [hconnection-0x744c309e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,796 DEBUG [hconnection-0x61eb9d5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,796 DEBUG [hconnection-0x42bb4f50-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,796 DEBUG [hconnection-0x71ec93d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,797 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54128, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,797 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:47,800 DEBUG [hconnection-0x22b1d81c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,800 DEBUG [hconnection-0x4e5c1b51-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:54:47,800 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54144, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,801 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54132, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,801 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54176, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,801 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54160, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,802 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54210, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,802 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54200, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,804 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54204, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,816 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54212, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:54:47,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:47,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:47,822 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:47,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:47,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:47,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:47,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:47,823 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,823 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:54:47,825 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:47,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:47,825 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:47,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:47,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:47,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,826 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,826 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:47,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:47,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:47,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:47,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:47,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:47,827 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:47,827 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:47,827 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:47,827 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,832 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@548e6ff6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:47,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:47,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:47,836 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,836 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:47,837 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:47,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:47,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:47,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:47,841 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:47,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:47,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:47,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:47,841 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:47,842 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,842 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:47,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,843 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:47,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:47,844 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,849 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:47,849 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:47,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:47,849 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:47,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:47,849 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:47,849 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,850 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:47,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:47,850 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:47,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507747845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507747847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507687843, max timestamp=1679507687849Num uniques 859; 2023-03-22 17:54:47,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507687843, max timestamp=1679507687849Num uniques 859; 2023-03-22 17:54:47,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:47,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:47,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507687843, max timestamp=1679507687849Num uniques 859; 2023-03-22 17:54:47,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:54:47,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507747867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507747871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507747871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/d635cc5156004b36aab3c1b33f8e5efe 2023-03-22 17:54:47,950 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:47,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:47,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:47,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507747952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507747952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:47,954 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:47,954 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:47,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:47,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:47,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,957 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:47,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:47,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,958 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:47,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:47,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:47,959 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:47,959 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:47,959 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:47,959 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:47,961 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:47,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:47,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:47,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75af90c9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:47,964 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:47,964 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,965 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,966 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:47,966 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:47,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:47,966 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:47,968 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:47,968 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,968 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:47,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:47,972 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:47,972 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:47,972 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:47,972 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:47,972 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:47,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507747972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,975 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:47,975 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,975 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:47,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:47,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:47,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507747975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:47,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507747976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,980 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:47,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:47,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:47,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:47,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:47,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:47,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:47,988 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:47,988 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:48,089 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:48,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:48,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:48,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:48,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:48,092 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:48,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:48,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:48,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:48,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:48,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:48,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:48,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:48,095 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:48,095 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:48,095 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:48,096 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:48,096 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:48,096 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:48,096 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:48,098 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a7df422[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:48,100 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:48,100 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,101 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:48,101 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,101 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:48,111 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:48,113 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:48,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:48,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:48,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:48,113 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:48,116 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,116 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,116 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:48,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,118 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,123 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:48,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:48,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:48,123 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:48,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:48,123 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:48,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:48,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:48,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507748154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507748157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507748176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507748179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507748179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,224 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:48,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:48,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:48,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:48,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:48,227 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:48,227 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:48,227 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:48,227 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:48,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,229 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:48,229 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:48,230 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,230 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:48,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:48,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:48,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:48,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:48,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:48,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:48,231 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:48,231 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:48,231 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:48,231 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:48,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:48,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:48,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:48,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:48,233 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@703a8f71[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:48,235 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:48,235 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:48,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:48,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:48,237 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,237 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:48,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:48,241 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:48,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:48,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:48,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:48,241 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:48,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:48,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:48,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,245 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:48,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:48,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,247 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:48,248 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:48,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:48,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:48,248 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:48,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:48,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:48,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:48,314 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/ce76ba4fc0d04eba995240449fc726d1 2023-03-22 17:54:48,354 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:48,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:48,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:48,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/3596b34bc076471fbf5ed2b372b507c4 2023-03-22 17:54:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:48,369 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:48,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:48,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:48,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:48,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/d635cc5156004b36aab3c1b33f8e5efe as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d635cc5156004b36aab3c1b33f8e5efe 2023-03-22 17:54:48,377 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,377 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:48,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:48,378 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,378 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d635cc5156004b36aab3c1b33f8e5efe, entries=150, sequenceid=13, filesize=11.6 K 2023-03-22 17:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:48,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:48,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/ce76ba4fc0d04eba995240449fc726d1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ce76ba4fc0d04eba995240449fc726d1 2023-03-22 17:54:48,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:48,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:48,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:48,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ce76ba4fc0d04eba995240449fc726d1, entries=150, sequenceid=13, filesize=11.6 K 2023-03-22 17:54:48,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/3596b34bc076471fbf5ed2b372b507c4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3596b34bc076471fbf5ed2b372b507c4 2023-03-22 17:54:48,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3596b34bc076471fbf5ed2b372b507c4, entries=150, sequenceid=13, filesize=11.6 K 2023-03-22 17:54:48,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=118.53 KB/121370 for a0a59fb2216419102dce85f7adab989d in 570ms, sequenceid=13, compaction requested=false 2023-03-22 17:54:48,394 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-22 17:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:48,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:48,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:48,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:48,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:48,396 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=118.53 KB heapSize=335.38 KB 2023-03-22 17:54:48,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:48,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:48,423 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/5ae697f6ff86417a830b7cd8aca5c66c 2023-03-22 17:54:48,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:48,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:48,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507748476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507748479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507748481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507748483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507748483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507748581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507748581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507748786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:48,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507748786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,871 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/83aafadd9fe74aa5aa6637f07e469ec0 2023-03-22 17:54:48,891 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/b10bb5feba0e49418a0b3fb61cb00626 2023-03-22 17:54:48,896 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/5ae697f6ff86417a830b7cd8aca5c66c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5ae697f6ff86417a830b7cd8aca5c66c 2023-03-22 17:54:48,901 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5ae697f6ff86417a830b7cd8aca5c66c, entries=150, sequenceid=47, filesize=11.6 K 2023-03-22 17:54:48,902 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/83aafadd9fe74aa5aa6637f07e469ec0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/83aafadd9fe74aa5aa6637f07e469ec0 2023-03-22 17:54:48,912 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/83aafadd9fe74aa5aa6637f07e469ec0, entries=150, sequenceid=47, filesize=11.6 K 2023-03-22 17:54:48,913 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/b10bb5feba0e49418a0b3fb61cb00626 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b10bb5feba0e49418a0b3fb61cb00626 2023-03-22 17:54:48,917 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b10bb5feba0e49418a0b3fb61cb00626, entries=150, sequenceid=47, filesize=11.6 K 2023-03-22 17:54:48,918 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~118.53 KB/121370, heapSize ~335.33 KB/343376, currentSize=60.38 KB/61830 for a0a59fb2216419102dce85f7adab989d in 522ms, sequenceid=47, compaction requested=false 2023-03-22 17:54:48,921 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:48,921 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:48,921 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:48,923 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:48,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576caf5f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:48,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:48,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,926 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:48,927 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:48,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:48,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:48,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:48,932 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:48,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:48,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:48,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:48,932 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:48,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:48,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:48,937 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,942 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:48,942 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:48,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:48,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:48,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:48,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:48,942 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,943 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:48,943 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:48,942 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:48,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:48,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:48,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:48,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:48,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:48,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:48,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-22 17:54:48,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:48,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:48,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:48,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:48,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:48,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:49,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507749006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507749007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507749008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,043 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:49,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:49,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:49,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:49,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:49,046 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:49,046 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,046 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:49,046 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:49,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,048 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:49,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:49,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:49,050 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,050 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:49,050 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:49,050 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:49,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:49,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e4c44a3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:49,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:49,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:49,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:49,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:49,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:49,062 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:49,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:49,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:49,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:49,062 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:49,063 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,063 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,064 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:49,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:49,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:49,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,071 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:49,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,071 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:49,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507749089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507749090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507749109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507749110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507749111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,171 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:49,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:49,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:49,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:49,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:49,174 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:49,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:49,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:49,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,176 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,176 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,177 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,177 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:49,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:49,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:49,178 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,178 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:49,178 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:49,178 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:49,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:49,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:49,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba567ac[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:49,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:49,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,183 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:49,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:49,187 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:49,202 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:49,202 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:49,202 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:49,202 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:49,202 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:49,203 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,203 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,203 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,204 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:49,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:49,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,209 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:49,209 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:49,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:49,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,309 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:49,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:49,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:49,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:49,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:49,313 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:49,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:49,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:49,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507749312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,315 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507749313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:49,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507749314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:49,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:49,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:49,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:49,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:49,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:49,321 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f403cae[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:49,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:49,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,325 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:49,326 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,326 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:49,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:49,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:49,330 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:49,330 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:49,330 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:49,331 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:49,331 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:49,332 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,332 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,332 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,335 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,341 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:49,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:49,341 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,341 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:49,341 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:49,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/7d9b4c783d5c4037aa088ed634c7d049 2023-03-22 17:54:49,442 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:49,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:49,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:49,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:49,445 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:49,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:49,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:49,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,447 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,448 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,448 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:49,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:49,449 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,449 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:49,449 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:49,449 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:49,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,450 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7de4744e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:49,452 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:49,453 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,454 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:49,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,456 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:49,458 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:49,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:49,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:49,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:49,459 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:49,460 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,460 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,460 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,461 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,462 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,466 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,466 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,466 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,466 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,466 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:49,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:49,466 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,466 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,467 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:49,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,467 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:49,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,567 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:49,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:49,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:49,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:49,570 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:49,570 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,570 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:49,570 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:49,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,573 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,573 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,574 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,574 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:49,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:49,575 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:49,575 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,575 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:49,575 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:49,575 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:49,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:49,577 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,577 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:49,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41315e0d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:49,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:49,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,581 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:49,581 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:49,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:49,582 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:49,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:49,585 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:49,585 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:49,585 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:49,585 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:49,585 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:49,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:49,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:49,592 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,592 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:49,592 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:49,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:49,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507749591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507749596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507749615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507749617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:49,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507749620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,692 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:49,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:49,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:49,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:49,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:49,696 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:49,696 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,696 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:49,697 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:49,697 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,697 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,699 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,699 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,699 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:49,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:49,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:49,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:49,708 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,708 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:49,708 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:49,708 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:49,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169c980[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:49,712 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:49,712 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,714 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:49,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:49,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:49,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:49,716 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:49,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:49,719 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:49,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:49,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:49,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:49,719 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:49,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,723 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,726 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:49,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:49,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:49,727 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:49,727 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:49,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/949def0e30d349448029433287aca24a 2023-03-22 17:54:49,827 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:49,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:49,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:49,830 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:49,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:49,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:49,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,832 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:49,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:49,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:49,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:49,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:49,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:49,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:49,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:49,835 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,835 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:49,835 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:49,835 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:49,837 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:49,839 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:49,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f82304f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:49,839 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,840 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:49,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,841 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:49,842 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,842 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:49,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:49,845 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:49,845 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:49,845 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:49,845 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:49,845 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:49,847 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,847 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,847 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:49,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:49,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,852 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:49,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,852 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:49,953 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:49,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:49,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:49,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:49,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:49,957 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:49,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:49,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:49,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,961 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,961 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,961 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,961 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:49,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:49,965 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,965 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:49,965 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:49,965 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:49,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@111a6bf5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:49,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:49,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:49,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:49,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:49,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:49,980 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:49,982 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:49,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:49,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:49,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:49,982 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:49,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:49,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:49,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:49,985 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:49,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:49,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:49,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:49,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:49,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:49,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:49,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:49,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:49,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:49,989 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:49,989 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:49,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:49,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:50,089 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:50,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:50,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:50,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:50,092 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:50,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:50,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:50,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:50,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:50,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:50,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:50,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:50,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:50,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:50,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:50,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:50,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:50,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:50,096 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,096 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:50,096 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:50,096 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:50,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:50,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:50,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,098 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a109eed[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:50,100 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:50,100 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:50,102 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:50,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,104 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:50,104 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,104 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:50,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:50,106 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:50,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:50,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:50,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:50,106 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:50,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,108 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,111 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:50,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:50,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:50,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:50,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:50,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:50,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,112 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:50,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:50,112 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:50,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:50,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507750120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507750123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507750126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,212 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:50,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:50,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:50,215 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:50,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:50,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:50,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:50,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,217 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:50,218 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:50,218 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,218 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:50,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:50,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:50,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:50,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:50,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:50,222 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:50,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15fa69e6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:50,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:50,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,224 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:50,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:50,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:50,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:50,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:50,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:50,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/651506b69bf846f8b90bed189f1ef42c 2023-03-22 17:54:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:50,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:50,229 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:50,229 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:50,229 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:50,229 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:50,229 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:50,232 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,232 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/7d9b4c783d5c4037aa088ed634c7d049 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/7d9b4c783d5c4037aa088ed634c7d049 2023-03-22 17:54:50,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,232 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,234 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/7d9b4c783d5c4037aa088ed634c7d049, entries=250, sequenceid=61, filesize=16.2 K 2023-03-22 17:54:50,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,237 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:50,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:50,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:50,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:50,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:50,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,244 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/949def0e30d349448029433287aca24a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/949def0e30d349448029433287aca24a 2023-03-22 17:54:50,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:50,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:50,244 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:50,245 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:50,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,245 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,245 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/949def0e30d349448029433287aca24a, entries=150, sequenceid=61, filesize=11.6 K 2023-03-22 17:54:50,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/651506b69bf846f8b90bed189f1ef42c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/651506b69bf846f8b90bed189f1ef42c 2023-03-22 17:54:50,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:50,266 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/651506b69bf846f8b90bed189f1ef42c, entries=150, sequenceid=61, filesize=11.6 K 2023-03-22 17:54:50,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a0a59fb2216419102dce85f7adab989d in 1280ms, sequenceid=61, compaction requested=true 2023-03-22 17:54:50,267 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:50,268 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:50,268 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:50,268 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:50,268 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:50,268 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:50,269 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:50,269 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:54:50,269 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,269 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d635cc5156004b36aab3c1b33f8e5efe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5ae697f6ff86417a830b7cd8aca5c66c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/7d9b4c783d5c4037aa088ed634c7d049] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=39.4 K 2023-03-22 17:54:50,269 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:50,269 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d635cc5156004b36aab3c1b33f8e5efe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679507687806 2023-03-22 17:54:50,269 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:54:50,269 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,269 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ce76ba4fc0d04eba995240449fc726d1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/83aafadd9fe74aa5aa6637f07e469ec0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/949def0e30d349448029433287aca24a] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.8 K 2023-03-22 17:54:50,269 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5ae697f6ff86417a830b7cd8aca5c66c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679507687862 2023-03-22 17:54:50,270 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ce76ba4fc0d04eba995240449fc726d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679507687806 2023-03-22 17:54:50,270 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 7d9b4c783d5c4037aa088ed634c7d049, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507688465 2023-03-22 17:54:50,270 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 83aafadd9fe74aa5aa6637f07e469ec0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679507687862 2023-03-22 17:54:50,270 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 949def0e30d349448029433287aca24a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507688468 2023-03-22 17:54:50,281 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:50,281 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:50,345 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:50,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:50,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:50,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:50,352 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:50,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:50,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:50,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:50,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:50,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:50,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:50,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:50,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:50,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:50,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:50,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:50,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:50,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:50,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:50,356 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-22 17:54:50,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:50,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:50,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:50,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:50,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:50,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:50,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:50,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507750613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507750614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,712 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/ca4e66c5315a40328b5dc630027895ad as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ca4e66c5315a40328b5dc630027895ad 2023-03-22 17:54:50,715 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/f700ef4f72bb40cd85188401911c2553 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/f700ef4f72bb40cd85188401911c2553 2023-03-22 17:54:50,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,717 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into ca4e66c5315a40328b5dc630027895ad(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:50,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507750715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,717 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:50,717 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507690268; duration=0sec 2023-03-22 17:54:50,717 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:50,718 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:50,719 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:50,719 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:54:50,719 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,719 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3596b34bc076471fbf5ed2b372b507c4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b10bb5feba0e49418a0b3fb61cb00626, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/651506b69bf846f8b90bed189f1ef42c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.8 K 2023-03-22 17:54:50,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507750718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,720 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3596b34bc076471fbf5ed2b372b507c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679507687806 2023-03-22 17:54:50,721 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b10bb5feba0e49418a0b3fb61cb00626, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679507687862 2023-03-22 17:54:50,721 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 651506b69bf846f8b90bed189f1ef42c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507688468 2023-03-22 17:54:50,721 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into f700ef4f72bb40cd85188401911c2553(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:50,721 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:50,722 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507690267; duration=0sec 2023-03-22 17:54:50,722 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:50,727 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#210 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:50,741 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/01147d9d2c024ca7a1594ae0adecf93f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/01147d9d2c024ca7a1594ae0adecf93f 2023-03-22 17:54:50,745 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into 01147d9d2c024ca7a1594ae0adecf93f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:50,745 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:50,745 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507690268; duration=0sec 2023-03-22 17:54:50,745 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:50,766 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/14f304a4bb9a46f1b80929d3dbb35a44 2023-03-22 17:54:50,779 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/479db439d8be419f9fa92bb488a98f41 2023-03-22 17:54:50,790 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/244c91248c1f475bbd4f1696538a6746 2023-03-22 17:54:50,793 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/14f304a4bb9a46f1b80929d3dbb35a44 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/14f304a4bb9a46f1b80929d3dbb35a44 2023-03-22 17:54:50,797 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/14f304a4bb9a46f1b80929d3dbb35a44, entries=150, sequenceid=79, filesize=11.6 K 2023-03-22 17:54:50,798 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/479db439d8be419f9fa92bb488a98f41 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/479db439d8be419f9fa92bb488a98f41 2023-03-22 17:54:50,801 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/479db439d8be419f9fa92bb488a98f41, entries=150, sequenceid=79, filesize=11.6 K 2023-03-22 17:54:50,804 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/244c91248c1f475bbd4f1696538a6746 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/244c91248c1f475bbd4f1696538a6746 2023-03-22 17:54:50,809 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/244c91248c1f475bbd4f1696538a6746, entries=150, sequenceid=79, filesize=11.6 K 2023-03-22 17:54:50,810 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a0a59fb2216419102dce85f7adab989d in 454ms, sequenceid=79, compaction requested=false 2023-03-22 17:54:50,810 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:50,810 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,810 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:50,810 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:50,810 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:50,813 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,814 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,814 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b24235e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:50,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:50,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,817 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:50,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:50,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:50,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:50,824 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:50,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:50,828 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:50,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:50,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:50,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:50,828 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:50,829 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,829 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,829 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:50,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:50,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:50,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:50,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:50,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:50,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,835 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:50,835 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:50,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:50,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:50,848 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-22 17:54:50,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:50,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-22 17:54:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:50,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:50,935 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/e8a03b15d12d4fad8199b5fea03b72c4 2023-03-22 17:54:50,935 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:50,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:50,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:50,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:50,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:50,938 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:50,938 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:50,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:50,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:50,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,940 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:50,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:50,940 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,941 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:50,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:50,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:50,942 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,942 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:50,942 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:50,942 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:50,945 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,945 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,945 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,945 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:50,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a3bc0c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:50,947 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:50,947 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,948 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:50,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:50,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507750948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:50,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507750950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:50,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:50,953 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:50,953 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:50,953 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:50,953 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:50,953 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:50,954 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,954 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:50,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,954 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:50,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:50,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:50,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:50,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:50,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:50,960 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:50,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:50,960 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:50,960 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:50,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:50,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:50,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507751051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507751052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,060 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:51,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:51,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:51,063 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:51,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:51,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:51,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:51,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:51,067 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,067 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:51,068 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:51,068 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:51,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:51,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@571aac23[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:51,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:51,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,072 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:51,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,073 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,075 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:51,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:51,077 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:51,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:51,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:51,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:51,077 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:51,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,080 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,081 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:51,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:51,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:51,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,083 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,084 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,084 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,083 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:51,084 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:51,084 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,084 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507751125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507751132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507751133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,184 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:51,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:51,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:51,187 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:51,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:51,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:51,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,189 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,189 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,189 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:51,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:51,191 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,191 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:51,191 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:51,191 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:51,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,194 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,194 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11694a7f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:51,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:51,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,198 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,198 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:51,198 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,198 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:51,198 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:51,198 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,202 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:51,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:51,205 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:51,205 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:51,205 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:51,205 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:51,205 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:51,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,211 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,211 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:51,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:51,211 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,211 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,211 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,212 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,212 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,212 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,212 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,212 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,212 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:51,212 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:51,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507751253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507751254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,312 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:51,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:51,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:51,316 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:51,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:51,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:51,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:51,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:51,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:51,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:51,320 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:51,322 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:51,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bbdb5df[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:51,325 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:51,325 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,326 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:51,327 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,327 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:51,328 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:51,333 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:51,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:51,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:51,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:51,334 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:51,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,336 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,337 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:51,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:51,341 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:51,341 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:51,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:51,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,341 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/8a0f2fd5b84d4d7da4cfd463644bf565 2023-03-22 17:54:51,441 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:51,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:51,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:51,444 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:51,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:51,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:51,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,446 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,447 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,447 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:51,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:51,448 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,448 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:51,448 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:51,448 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:51,449 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:51,450 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:51,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fea2578[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:51,452 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:51,452 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,453 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:51,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:51,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:51,454 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:51,455 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:51,458 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:51,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:51,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:51,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:51,458 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:51,460 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,460 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,460 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,461 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,462 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:51,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:51,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:51,465 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:51,465 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507751557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:51,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507751558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,565 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:51,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:51,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:51,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:51,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:51,568 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:51,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:51,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:51,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,569 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,569 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,570 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,570 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:51,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:51,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:51,571 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:51,571 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,571 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:51,571 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:51,571 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:51,571 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,571 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:51,572 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:51,572 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:51,573 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,573 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,573 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,574 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:51,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d13e712[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:51,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:51,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,577 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:51,578 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:51,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:51,579 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,579 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:51,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:51,581 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:51,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:51,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:51,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:51,582 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:51,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,594 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,595 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:51,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:51,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,595 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,595 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:51,695 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:51,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:51,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:51,698 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:51,698 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,698 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:51,698 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:51,699 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,699 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,700 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:51,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:51,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:51,702 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,702 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:51,702 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:51,702 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:51,703 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,703 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,704 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,704 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f2727d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:51,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:51,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,706 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,707 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,707 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:51,709 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:51,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:51,711 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:51,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:51,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:51,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:51,711 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:51,713 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,713 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:51,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,713 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:51,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,717 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:51,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:51,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,717 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:51,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:51,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:51,717 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:51,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/32ac4d33ce844de4b45057c36c4a4a21 2023-03-22 17:54:51,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/e8a03b15d12d4fad8199b5fea03b72c4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e8a03b15d12d4fad8199b5fea03b72c4 2023-03-22 17:54:51,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e8a03b15d12d4fad8199b5fea03b72c4, entries=150, sequenceid=97, filesize=11.6 K 2023-03-22 17:54:51,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/8a0f2fd5b84d4d7da4cfd463644bf565 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/8a0f2fd5b84d4d7da4cfd463644bf565 2023-03-22 17:54:51,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/8a0f2fd5b84d4d7da4cfd463644bf565, entries=150, sequenceid=97, filesize=11.6 K 2023-03-22 17:54:51,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/32ac4d33ce844de4b45057c36c4a4a21 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/32ac4d33ce844de4b45057c36c4a4a21 2023-03-22 17:54:51,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/32ac4d33ce844de4b45057c36c4a4a21, entries=150, sequenceid=97, filesize=11.6 K 2023-03-22 17:54:51,812 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a0a59fb2216419102dce85f7adab989d in 891ms, sequenceid=97, compaction requested=true 2023-03-22 17:54:51,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:51,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:51,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:51,813 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:51,813 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:51,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:51,814 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:51,814 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:54:51,814 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,814 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/f700ef4f72bb40cd85188401911c2553, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/14f304a4bb9a46f1b80929d3dbb35a44, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e8a03b15d12d4fad8199b5fea03b72c4] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.9 K 2023-03-22 17:54:51,815 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:51,815 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:54:51,815 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,815 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ca4e66c5315a40328b5dc630027895ad, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/479db439d8be419f9fa92bb488a98f41, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/8a0f2fd5b84d4d7da4cfd463644bf565] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.9 K 2023-03-22 17:54:51,815 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f700ef4f72bb40cd85188401911c2553, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507688468 2023-03-22 17:54:51,815 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ca4e66c5315a40328b5dc630027895ad, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507688468 2023-03-22 17:54:51,816 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 14f304a4bb9a46f1b80929d3dbb35a44, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1679507689005 2023-03-22 17:54:51,816 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 479db439d8be419f9fa92bb488a98f41, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1679507689005 2023-03-22 17:54:51,816 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e8a03b15d12d4fad8199b5fea03b72c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507690607 2023-03-22 17:54:51,816 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 8a0f2fd5b84d4d7da4cfd463644bf565, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507690607 2023-03-22 17:54:51,818 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:51,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:51,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:51,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:51,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:51,823 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:51,823 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,823 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:51,823 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:51,824 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,824 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,825 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,825 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:51,825 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:51,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:51,826 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:51,826 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:51,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:51,829 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:51,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:51,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:51,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:51,839 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:51,840 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:51,841 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-22 17:54:51,841 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:51,841 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:51,841 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:51,841 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:51,841 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:51,841 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:51,854 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/bd172b711bdc4de98c5065b2b537cbf0 2023-03-22 17:54:51,864 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/bd67ea28784b4a4381f0a8b79497d97d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd67ea28784b4a4381f0a8b79497d97d 2023-03-22 17:54:51,869 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into bd67ea28784b4a4381f0a8b79497d97d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:51,870 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:51,870 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507691813; duration=0sec 2023-03-22 17:54:51,870 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:51,870 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:51,871 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:51,871 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:54:51,871 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:51,871 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/01147d9d2c024ca7a1594ae0adecf93f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/244c91248c1f475bbd4f1696538a6746, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/32ac4d33ce844de4b45057c36c4a4a21] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.9 K 2023-03-22 17:54:51,872 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 01147d9d2c024ca7a1594ae0adecf93f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507688468 2023-03-22 17:54:51,872 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 244c91248c1f475bbd4f1696538a6746, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1679507689005 2023-03-22 17:54:51,872 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 32ac4d33ce844de4b45057c36c4a4a21, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507690607 2023-03-22 17:54:51,888 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/ed5ce508bffa42968d453f7f65e47059 2023-03-22 17:54:51,891 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#220 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:51,907 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/310429eddd7b419084d9e6dd36c6eef5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/310429eddd7b419084d9e6dd36c6eef5 2023-03-22 17:54:51,912 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into 310429eddd7b419084d9e6dd36c6eef5(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:51,912 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:51,912 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507691813; duration=0sec 2023-03-22 17:54:51,912 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:52,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:52,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:52,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507752080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507752081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507752183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507752183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,242 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/15b9eff3edd54f18b69db0285feb4b87 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/15b9eff3edd54f18b69db0285feb4b87 2023-03-22 17:54:52,246 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into 15b9eff3edd54f18b69db0285feb4b87(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:52,246 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:52,246 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507691813; duration=0sec 2023-03-22 17:54:52,246 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:52,317 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/3ea7624175454c4ca6684976ba13736e 2023-03-22 17:54:52,322 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/bd172b711bdc4de98c5065b2b537cbf0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd172b711bdc4de98c5065b2b537cbf0 2023-03-22 17:54:52,327 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd172b711bdc4de98c5065b2b537cbf0, entries=150, sequenceid=114, filesize=11.6 K 2023-03-22 17:54:52,331 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/ed5ce508bffa42968d453f7f65e47059 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ed5ce508bffa42968d453f7f65e47059 2023-03-22 17:54:52,337 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ed5ce508bffa42968d453f7f65e47059, entries=150, sequenceid=114, filesize=11.6 K 2023-03-22 17:54:52,340 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/3ea7624175454c4ca6684976ba13736e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3ea7624175454c4ca6684976ba13736e 2023-03-22 17:54:52,345 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3ea7624175454c4ca6684976ba13736e, entries=150, sequenceid=114, filesize=11.6 K 2023-03-22 17:54:52,346 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a0a59fb2216419102dce85f7adab989d in 505ms, sequenceid=114, compaction requested=false 2023-03-22 17:54:52,346 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:52,346 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:52,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:52,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:52,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:52,349 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:52,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20772a31[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:52,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:52,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,353 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:52,353 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,353 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:52,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,354 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:52,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:52,370 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:52,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:52,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:52,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:52,370 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:52,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,374 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,374 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,376 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:52,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:52,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:52,384 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,384 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:52,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:52,390 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-22 17:54:52,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:52,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:52,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:52,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:52,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:52,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:52,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507752411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507752414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/da1e67c88fff458389641fdf150696a2 2023-03-22 17:54:52,485 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:52,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:52,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:52,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:52,490 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:52,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:52,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:52,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:52,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:52,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:52,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:52,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:52,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:52,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:52,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,502 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:52,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e881aa2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:52,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:52,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:52,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:52,513 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,513 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:52,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:52,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507752515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507752517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:52,520 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:52,520 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:52,520 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:52,520 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:52,520 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:52,522 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,522 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,522 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,523 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,530 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:52,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,530 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:52,530 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:52,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:52,631 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:52,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:52,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:52,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:52,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:52,634 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:52,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:52,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:52,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:52,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,636 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,637 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,637 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:52,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:52,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:52,638 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,638 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:52,638 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:52,638 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:52,641 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,641 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,641 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,641 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cd15516[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:52,643 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:52,643 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:52,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,646 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:52,648 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:52,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:52,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:52,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:52,648 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:52,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,651 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,651 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,651 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,653 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:52,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:52,655 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,656 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:52,656 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:52,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507752718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:52,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507752722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,756 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:52,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:52,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:52,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:52,759 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:52,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:52,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:52,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:52,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,761 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,761 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,761 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:52,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:52,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:52,763 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:52,763 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:52,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4915b36b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:52,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:52,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:52,769 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,769 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:52,770 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,771 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:52,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:52,773 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:52,773 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:52,773 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:52,773 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:52,773 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:52,775 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,775 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,775 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,775 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,776 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,779 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:52,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:52,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,779 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:52,780 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:52,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/3125366a55434b0fa811905f71efd885 2023-03-22 17:54:52,880 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:52,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:52,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:52,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:52,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:52,883 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:52,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:52,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:52,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:52,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,885 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:52,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:52,888 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,888 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:52,888 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:52,888 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:52,890 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,890 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,890 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,890 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:52,891 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@502a3eda[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:52,892 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:52,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:52,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:52,897 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:52,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:52,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:52,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:52,897 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:52,899 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,899 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:52,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,899 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,900 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:52,901 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,903 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,903 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:52,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:52,903 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:52,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:52,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:52,903 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:52,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:52,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:52,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:52,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:52,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:52,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:52,904 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:52,904 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:53,004 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:53,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:53,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:53,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:53,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:53,014 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:53,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:53,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:53,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:53,015 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,015 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,019 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:53,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:53,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:53,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:53,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507753020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,026 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:53,026 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:53,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507753024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,028 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:53,028 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:53,028 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:53,028 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:53,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:53,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:53,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:53,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:53,030 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:53,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:53,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ea96377[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:53,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:53,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,034 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,034 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:53,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:53,037 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,037 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:53,037 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:53,037 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:53,040 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:53,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:53,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:53,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:53,040 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:53,042 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,042 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:53,042 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:53,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:53,043 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:53,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:53,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:53,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:53,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:53,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:53,055 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:53,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:53,055 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:53,055 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:53,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507753131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,135 DEBUG [Thread-1629] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:53,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507753149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,153 DEBUG [Thread-1631] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:53,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507753153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,156 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:53,156 DEBUG [Thread-1633] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:53,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:53,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:53,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:53,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:53,160 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:53,160 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:53,160 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:53,160 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:53,160 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,160 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,161 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,161 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:53,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:53,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:53,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:53,163 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:53,164 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:53,164 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:53,164 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:53,172 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68b6ee51[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:53,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:53,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,175 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:53,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:53,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:53,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:53,180 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:53,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:53,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:53,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:53,180 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:53,182 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,182 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:53,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,182 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:53,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:53,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:53,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:53,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:53,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:53,190 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:53,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:53,190 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,190 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,190 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:53,190 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:53,190 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,190 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:53,190 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:53,244 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/728e5fb148bf49bb8d9a9b9111aa9e63 2023-03-22 17:54:53,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/da1e67c88fff458389641fdf150696a2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/da1e67c88fff458389641fdf150696a2 2023-03-22 17:54:53,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/da1e67c88fff458389641fdf150696a2, entries=150, sequenceid=134, filesize=11.8 K 2023-03-22 17:54:53,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/3125366a55434b0fa811905f71efd885 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/3125366a55434b0fa811905f71efd885 2023-03-22 17:54:53,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/3125366a55434b0fa811905f71efd885, entries=150, sequenceid=134, filesize=11.8 K 2023-03-22 17:54:53,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/728e5fb148bf49bb8d9a9b9111aa9e63 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/728e5fb148bf49bb8d9a9b9111aa9e63 2023-03-22 17:54:53,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/728e5fb148bf49bb8d9a9b9111aa9e63, entries=150, sequenceid=134, filesize=11.8 K 2023-03-22 17:54:53,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a0a59fb2216419102dce85f7adab989d in 872ms, sequenceid=134, compaction requested=true 2023-03-22 17:54:53,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:53,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:53,262 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:53,262 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:53,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:53,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:53,263 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:53,263 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:54:53,263 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:53,263 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:53,263 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:54:53,263 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd67ea28784b4a4381f0a8b79497d97d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd172b711bdc4de98c5065b2b537cbf0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/da1e67c88fff458389641fdf150696a2] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=35.2 K 2023-03-22 17:54:53,263 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:53,263 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/15b9eff3edd54f18b69db0285feb4b87, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ed5ce508bffa42968d453f7f65e47059, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/3125366a55434b0fa811905f71efd885] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=35.2 K 2023-03-22 17:54:53,264 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting bd67ea28784b4a4381f0a8b79497d97d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507690607 2023-03-22 17:54:53,264 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 15b9eff3edd54f18b69db0285feb4b87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507690607 2023-03-22 17:54:53,264 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting bd172b711bdc4de98c5065b2b537cbf0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507690935 2023-03-22 17:54:53,264 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ed5ce508bffa42968d453f7f65e47059, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507690935 2023-03-22 17:54:53,264 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting da1e67c88fff458389641fdf150696a2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507692079 2023-03-22 17:54:53,265 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3125366a55434b0fa811905f71efd885, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507692079 2023-03-22 17:54:53,271 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:53,271 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:53,290 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:53,291 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1f5de1db040e42eb8edcc8ba606c9b3b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f5de1db040e42eb8edcc8ba606c9b3b 2023-03-22 17:54:53,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:53,293 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/0fcb8681c19f4b4481e19efdf178a2e3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/0fcb8681c19f4b4481e19efdf178a2e3 2023-03-22 17:54:53,298 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into 1f5de1db040e42eb8edcc8ba606c9b3b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:53,298 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:53,298 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507693262; duration=0sec 2023-03-22 17:54:53,298 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:53,298 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:53,300 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:53,300 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:54:53,300 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:53,300 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/310429eddd7b419084d9e6dd36c6eef5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3ea7624175454c4ca6684976ba13736e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/728e5fb148bf49bb8d9a9b9111aa9e63] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=35.2 K 2023-03-22 17:54:53,301 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 310429eddd7b419084d9e6dd36c6eef5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507690607 2023-03-22 17:54:53,301 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3ea7624175454c4ca6684976ba13736e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679507690935 2023-03-22 17:54:53,301 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 728e5fb148bf49bb8d9a9b9111aa9e63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507692079 2023-03-22 17:54:53,302 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into 0fcb8681c19f4b4481e19efdf178a2e3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:53,302 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:53,302 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507693262; duration=0sec 2023-03-22 17:54:53,302 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:53,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:53,332 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:53,332 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:53,332 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:53,332 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:53,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,334 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:53,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:53,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,335 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,335 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:53,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:53,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:53,338 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:53,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:53,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:53,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:53,396 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-22 17:54:53,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:53,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:53,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:53,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:53,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:53,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:53,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:53,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:53,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507753547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507753551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507753651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507753655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,825 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/b1b3966604f54345a741cc4d5eae4960 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1b3966604f54345a741cc4d5eae4960 2023-03-22 17:54:53,829 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into b1b3966604f54345a741cc4d5eae4960(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:53,829 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:53,829 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507693262; duration=0sec 2023-03-22 17:54:53,830 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:53,833 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/c53c8ab8f30d4dbca565a4933062e994 2023-03-22 17:54:53,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507753856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:53,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:53,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507753859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:54,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507754159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:54,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507754161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,245 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1e357d1ec2f54d2dad7228e36315c65f 2023-03-22 17:54:54,662 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/8e44e9851f1942a2b7be37aed3f22a03 2023-03-22 17:54:54,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:54,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:54,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507754664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,666 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/c53c8ab8f30d4dbca565a4933062e994 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c53c8ab8f30d4dbca565a4933062e994 2023-03-22 17:54:54,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507754665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,670 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c53c8ab8f30d4dbca565a4933062e994, entries=150, sequenceid=151, filesize=11.8 K 2023-03-22 17:54:54,671 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1e357d1ec2f54d2dad7228e36315c65f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1e357d1ec2f54d2dad7228e36315c65f 2023-03-22 17:54:54,674 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1e357d1ec2f54d2dad7228e36315c65f, entries=150, sequenceid=151, filesize=11.8 K 2023-03-22 17:54:54,675 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/8e44e9851f1942a2b7be37aed3f22a03 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/8e44e9851f1942a2b7be37aed3f22a03 2023-03-22 17:54:54,679 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/8e44e9851f1942a2b7be37aed3f22a03, entries=150, sequenceid=151, filesize=11.8 K 2023-03-22 17:54:54,679 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a0a59fb2216419102dce85f7adab989d in 1283ms, sequenceid=151, compaction requested=false 2023-03-22 17:54:54,680 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:54,680 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:54,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cdc96c7[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:54,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:54,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:54,685 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:54,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:54,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:54,685 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,685 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:54,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:54,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:54,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:54,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:54,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:54,690 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:54,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:54,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:54,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:54,690 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:54,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:54,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:54,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:54,692 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:54,693 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,695 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:54,695 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:54,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:54,695 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,696 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:54,696 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:54,696 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,696 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:54,696 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:54,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:54,696 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:54,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:54,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:54,696 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:54,696 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:54,696 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:54,796 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:54,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:54,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:54,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:54,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:54,801 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:54,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:54,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:54,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:54,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:54,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:54,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:54,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:54,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:54,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:54,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:54,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:54,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:54,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:54,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:54,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:54,808 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:54,808 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:54,809 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-22 17:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:54,818 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/444b124fcb2c4bd89519aedeb463333f 2023-03-22 17:54:55,233 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/32abf14ec1a54b94ad674d94b5dab943 2023-03-22 17:54:55,660 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/38543affcb574400b5e8b097e41abab4 2023-03-22 17:54:55,664 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/444b124fcb2c4bd89519aedeb463333f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/444b124fcb2c4bd89519aedeb463333f 2023-03-22 17:54:55,667 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/444b124fcb2c4bd89519aedeb463333f, entries=150, sequenceid=169, filesize=11.8 K 2023-03-22 17:54:55,668 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/32abf14ec1a54b94ad674d94b5dab943 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/32abf14ec1a54b94ad674d94b5dab943 2023-03-22 17:54:55,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:55,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:55,672 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/32abf14ec1a54b94ad674d94b5dab943, entries=150, sequenceid=169, filesize=11.8 K 2023-03-22 17:54:55,673 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/38543affcb574400b5e8b097e41abab4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/38543affcb574400b5e8b097e41abab4 2023-03-22 17:54:55,676 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/38543affcb574400b5e8b097e41abab4, entries=150, sequenceid=169, filesize=11.8 K 2023-03-22 17:54:55,677 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=33.54 KB/34350 for a0a59fb2216419102dce85f7adab989d in 869ms, sequenceid=169, compaction requested=true 2023-03-22 17:54:55,677 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:55,677 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:55,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:55,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:55,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:55,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:55,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:55,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:54:55,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:55,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:55,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:55,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:55,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:55,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:55,684 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d01a568[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:55,686 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:55,686 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:55,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:55,687 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,687 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:55,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:55,689 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:55,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:55,691 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:55,691 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:55,691 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:55,691 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:55,691 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:55,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,693 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:55,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,694 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:55,697 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:55,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:55,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:55,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,698 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:55,698 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:55,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507695711, max timestamp=1679507695741Num uniques 859; 2023-03-22 17:54:55,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:54:55,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507695711, max timestamp=1679507695741Num uniques 859; 2023-03-22 17:54:55,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:54:55,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679507695711, max timestamp=1679507695741Num uniques 859; 2023-03-22 17:54:55,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-22 17:54:55,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507755772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:55,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507755774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,798 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:55,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:55,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:55,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:55,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:55,801 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:55,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:55,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:55,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:55,802 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,802 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:55,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:55,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:55,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:55,805 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:55,805 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:55,805 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:55,805 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:55,807 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,807 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,807 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,807 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4876cd64[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:55,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:55,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,811 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:55,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:55,811 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:55,811 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,811 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:55,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:55,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:55,812 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:55,814 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:55,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:55,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:55,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:55,815 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:55,816 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,816 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,817 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:55,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:55,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:55,821 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,821 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:55,821 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:55,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:55,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:55,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:55,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:55,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:55,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54128 deadline: 1679507755876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507755877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,921 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:55,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:55,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:55,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:55,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:55,928 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:55,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:55,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:55,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:55,929 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,929 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:55,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:55,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:55,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:55,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:55,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:55,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:55,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:55,933 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:55,933 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:55,933 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:55,933 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:55,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:55,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ed765d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:55,937 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:55,937 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,938 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,938 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:55,938 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:55,938 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:55,938 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,938 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:55,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:55,943 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:55,943 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:55,944 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:55,944 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:55,944 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:55,946 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:55,947 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,947 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,947 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:55,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:55,951 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:55,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:55,951 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:55,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:55,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:55,951 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:55,952 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:55,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,052 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:56,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:56,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:56,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:56,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:56,055 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:56,055 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,055 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:56,055 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:56,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,057 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,057 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:56,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:56,059 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,059 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:56,060 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:56,060 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:56,061 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:56,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6fa0c9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:56,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:56,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:56,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:56,070 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,070 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:56,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:56,072 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:56,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:56,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:56,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:56,072 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:56,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,076 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:56,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:56,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:56,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,078 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,078 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:56,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,079 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:56,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:56,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54128 deadline: 1679507756079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:56,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507756081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/00ab464897594460b92b04d8df6979ec 2023-03-22 17:54:56,179 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:56,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:56,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:56,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:56,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:56,182 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:56,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:56,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:56,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:56,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:56,188 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,188 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f4560a9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:56,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:56,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:56,191 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,192 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:56,193 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,193 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:56,193 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,193 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:56,195 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:56,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:56,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:56,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:56,196 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:56,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,203 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,204 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:56,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:56,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,207 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:56,207 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,207 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,307 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:56,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:56,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:56,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:56,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:56,310 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:56,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:56,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:56,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:56,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:56,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:56,315 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:56,315 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,315 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:56,315 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:56,315 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:56,315 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,316 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:56,316 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:56,316 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:56,319 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,319 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,319 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,319 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:56,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1269fc8[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:56,322 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:56,322 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,324 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,325 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,325 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:56,325 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:56,325 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:56,325 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:56,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:56,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,326 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,326 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:56,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:56,329 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:56,329 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:56,330 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:56,330 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:56,330 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:56,332 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,332 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,332 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,341 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,341 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,341 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:56,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:56,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:56,342 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,342 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:56,342 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,342 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,342 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,342 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,342 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,342 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:56,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507756384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:56,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54128 deadline: 1679507756385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,442 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:56,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:56,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:56,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:56,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:56,445 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:56,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:56,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:56,446 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,446 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,448 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,448 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,448 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,448 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:56,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:56,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:56,450 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,450 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:56,450 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:56,450 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:56,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:56,451 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d855e2f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:56,453 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:56,453 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,455 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:56,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:56,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:56,455 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:56,456 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:56,459 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:56,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:56,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:56,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:56,459 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:56,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,472 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,472 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:56,472 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,472 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,473 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,473 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,473 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,473 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,473 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:56,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:56,476 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:56,476 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:56,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/57d00dbbc539405cb0eabb8c6b79c847 2023-03-22 17:54:56,576 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:56,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:56,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:56,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:56,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:56,579 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:56,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:56,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:56,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,580 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,580 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:56,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:56,582 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,582 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:56,582 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:56,582 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:56,584 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:56,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7feec0be[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:56,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:56,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:56,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:56,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:56,593 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:56,595 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:56,595 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:56,595 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:56,595 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:56,595 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:56,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,600 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:56,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:56,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:56,601 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:56,601 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:56,701 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:56,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:56,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:56,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:56,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:56,705 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:56,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:56,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:56,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,707 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,707 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,708 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,708 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:56,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:56,709 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:56,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:56,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:56,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,712 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:56,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:56,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bfab7c2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:56,715 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:56,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:56,716 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:56,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,717 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:56,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:56,719 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:56,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:56,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:56,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:56,719 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:56,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,723 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,728 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,728 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,729 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:56,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:56,729 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:56,729 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:56,830 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:56,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:56,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:56,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:56,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:56,833 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:56,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:56,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:56,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:56,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:56,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:56,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:56,840 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,840 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:56,840 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:56,840 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:56,842 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,842 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,842 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:56,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e17a5aa[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:56,844 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:56,844 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,846 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:56,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:56,848 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:56,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:56,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:56,850 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:56,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:56,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:56,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:56,851 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:56,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:56,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,860 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,860 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,860 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:56,860 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:56,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:56,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:56,860 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,861 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:56,861 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:56,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:56,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:56,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:56,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54128 deadline: 1679507756889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:56,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507756890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/430fa73eca644357a6a36674ba1ff19a 2023-03-22 17:54:56,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/00ab464897594460b92b04d8df6979ec as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/00ab464897594460b92b04d8df6979ec 2023-03-22 17:54:56,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/00ab464897594460b92b04d8df6979ec, entries=150, sequenceid=179, filesize=11.8 K 2023-03-22 17:54:56,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/57d00dbbc539405cb0eabb8c6b79c847 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/57d00dbbc539405cb0eabb8c6b79c847 2023-03-22 17:54:56,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/57d00dbbc539405cb0eabb8c6b79c847, entries=150, sequenceid=179, filesize=11.8 K 2023-03-22 17:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/430fa73eca644357a6a36674ba1ff19a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430fa73eca644357a6a36674ba1ff19a 2023-03-22 17:54:56,944 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430fa73eca644357a6a36674ba1ff19a, entries=150, sequenceid=179, filesize=11.8 K 2023-03-22 17:54:56,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.78 KB/136995 for a0a59fb2216419102dce85f7adab989d in 1264ms, sequenceid=179, compaction requested=true 2023-03-22 17:54:56,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:56,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:56,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:56,944 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:54:56,945 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:54:56,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:56,946 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:54:56,946 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:54:56,946 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:54:56,946 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,946 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:54:56,946 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/0fcb8681c19f4b4481e19efdf178a2e3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c53c8ab8f30d4dbca565a4933062e994, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/444b124fcb2c4bd89519aedeb463333f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/00ab464897594460b92b04d8df6979ec] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=47.3 K 2023-03-22 17:54:56,946 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,946 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f5de1db040e42eb8edcc8ba606c9b3b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1e357d1ec2f54d2dad7228e36315c65f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/32abf14ec1a54b94ad674d94b5dab943, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/57d00dbbc539405cb0eabb8c6b79c847] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=47.3 K 2023-03-22 17:54:56,946 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0fcb8681c19f4b4481e19efdf178a2e3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507692079 2023-03-22 17:54:56,946 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1f5de1db040e42eb8edcc8ba606c9b3b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507692079 2023-03-22 17:54:56,947 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c53c8ab8f30d4dbca565a4933062e994, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1679507692406 2023-03-22 17:54:56,947 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1e357d1ec2f54d2dad7228e36315c65f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1679507692406 2023-03-22 17:54:56,947 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 444b124fcb2c4bd89519aedeb463333f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507693540 2023-03-22 17:54:56,947 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 32abf14ec1a54b94ad674d94b5dab943, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507693540 2023-03-22 17:54:56,947 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 00ab464897594460b92b04d8df6979ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679507695676 2023-03-22 17:54:56,947 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 57d00dbbc539405cb0eabb8c6b79c847, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679507695676 2023-03-22 17:54:56,954 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#237 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:56,956 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#238 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:56,961 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:56,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:56,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:56,964 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/e64833f523064b13a6364b204f6ec0f3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e64833f523064b13a6364b204f6ec0f3 2023-03-22 17:54:56,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:56,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:56,965 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:56,965 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,965 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:56,965 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:56,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,967 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/7cc6441ac53548528f3c2f1c303bd750 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/7cc6441ac53548528f3c2f1c303bd750 2023-03-22 17:54:56,967 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:56,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:56,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:56,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:56,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:56,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:56,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:56,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:56,969 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:56,969 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:56,969 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:56,970 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:56,970 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:56,970 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,970 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:56,970 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=133.78 KB heapSize=384.07 KB 2023-03-22 17:54:56,970 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into e64833f523064b13a6364b204f6ec0f3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:56,970 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:56,970 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:56,970 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=12, startTime=1679507696944; duration=0sec 2023-03-22 17:54:56,970 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:56,970 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:56,970 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:56,971 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:54:56,971 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:56,971 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:56,971 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:56,972 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:54:56,972 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:54:56,972 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:56,972 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1b3966604f54345a741cc4d5eae4960, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/8e44e9851f1942a2b7be37aed3f22a03, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/38543affcb574400b5e8b097e41abab4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430fa73eca644357a6a36674ba1ff19a] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=47.3 K 2023-03-22 17:54:56,973 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b1b3966604f54345a741cc4d5eae4960, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679507692079 2023-03-22 17:54:56,973 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 8e44e9851f1942a2b7be37aed3f22a03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1679507692406 2023-03-22 17:54:56,973 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into 7cc6441ac53548528f3c2f1c303bd750(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:56,973 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:56,973 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=12, startTime=1679507696944; duration=0sec 2023-03-22 17:54:56,973 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:56,973 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 38543affcb574400b5e8b097e41abab4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507693540 2023-03-22 17:54:56,974 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 430fa73eca644357a6a36674ba1ff19a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679507695676 2023-03-22 17:54:56,981 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#240 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:56,985 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/35fca7f94e8e407da4720cb2def34afd 2023-03-22 17:54:56,990 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/60e12e43cf1b40798631cfbc43fa8b9a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/60e12e43cf1b40798631cfbc43fa8b9a 2023-03-22 17:54:56,996 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into 60e12e43cf1b40798631cfbc43fa8b9a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:56,996 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:56,996 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=12, startTime=1679507696944; duration=0sec 2023-03-22 17:54:56,996 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:57,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:57,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:57,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507757185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507757188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507757191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,194 DEBUG [Thread-1633] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:57,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507757289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507757291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,398 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/f9d8cbeb899d43628159b79d0dcd807d 2023-03-22 17:54:57,413 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/b1def152e4b24e789cba417c08641d10 2023-03-22 17:54:57,418 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/35fca7f94e8e407da4720cb2def34afd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/35fca7f94e8e407da4720cb2def34afd 2023-03-22 17:54:57,422 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/35fca7f94e8e407da4720cb2def34afd, entries=150, sequenceid=217, filesize=11.8 K 2023-03-22 17:54:57,423 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/f9d8cbeb899d43628159b79d0dcd807d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/f9d8cbeb899d43628159b79d0dcd807d 2023-03-22 17:54:57,426 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/f9d8cbeb899d43628159b79d0dcd807d, entries=150, sequenceid=217, filesize=11.8 K 2023-03-22 17:54:57,427 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/b1def152e4b24e789cba417c08641d10 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1def152e4b24e789cba417c08641d10 2023-03-22 17:54:57,431 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1def152e4b24e789cba417c08641d10, entries=150, sequenceid=217, filesize=11.8 K 2023-03-22 17:54:57,432 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~133.78 KB/136995, heapSize ~384.02 KB/393240, currentSize=46.96 KB/48090 for a0a59fb2216419102dce85f7adab989d in 462ms, sequenceid=217, compaction requested=false 2023-03-22 17:54:57,432 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:57,432 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:57,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49b4e461[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:57,443 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:57,443 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,445 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:57,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:57,448 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:57,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,448 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:57,452 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:57,452 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:57,452 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:57,452 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:57,452 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:57,454 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,454 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,454 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:57,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:57,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,459 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,459 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,459 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:57,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:57,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:54:57,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:57,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:57,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:57,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:57,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:57,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:57,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507697514, max timestamp=1679507697528Num uniques 859; 2023-03-22 17:54:57,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507697514, max timestamp=1679507697528Num uniques 859; 2023-03-22 17:54:57,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:57,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507697514, max timestamp=1679507697528Num uniques 859; 2023-03-22 17:54:57,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:57,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:57,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507757550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507757554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,559 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:57,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:57,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:57,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:57,562 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:57,562 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:57,562 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:57,562 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:57,562 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,562 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,564 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:57,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:57,565 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,565 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:57,565 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:57,566 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:57,568 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40d636d9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:57,569 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:57,570 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,571 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:57,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:57,572 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:57,573 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,573 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:57,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:57,575 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:57,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:57,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:57,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:57,575 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:57,577 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,577 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,577 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,577 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,578 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:57,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:57,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:57,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,584 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,584 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,584 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,584 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,584 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,584 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:57,584 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:57,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507757655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507757659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,684 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:57,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:57,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:57,687 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:57,687 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:57,687 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:57,687 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:57,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,689 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,689 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,689 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,689 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:57,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:57,691 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,691 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:57,691 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:57,691 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:57,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eeccda6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:57,694 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:57,694 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,695 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:57,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:57,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:57,698 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:57,700 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:57,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:57,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:57,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:57,700 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:57,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,703 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,705 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,705 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,705 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,705 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:57,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,705 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:57,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:57,705 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,705 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,706 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,706 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:57,706 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,706 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:57,706 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,806 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:57,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:57,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:57,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:57,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:57,809 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:57,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:57,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:57,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:57,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,814 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:57,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:57,815 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,815 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:57,816 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:57,816 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:57,817 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:57,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1df51524[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:57,819 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:57,819 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,820 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,820 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:57,820 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:57,820 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:57,820 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,820 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:57,822 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:57,822 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,822 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,822 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:57,825 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:57,825 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:57,825 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:57,825 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:57,825 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:57,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,828 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,832 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,833 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,833 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,833 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,833 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:57,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:57,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,833 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,834 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:57,834 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:57,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:57,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507757857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507757861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507757896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/5a3dcae63d524c7a96f81ebf8ef69538 2023-03-22 17:54:57,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:57,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54128 deadline: 1679507757899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/2d6f14bca789452db9ca49d6724fe1dc 2023-03-22 17:54:57,934 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:57,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:57,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:57,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:57,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:57,952 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:57,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:57,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:57,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:57,952 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,953 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,954 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,954 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:57,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:57,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:57,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:57,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:57,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:57,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:57,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738c7d17[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:57,962 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:57,962 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,965 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:57,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:57,967 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:57,970 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:57,970 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:57,970 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:57,970 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:57,970 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:57,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:57,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:57,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,974 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:57,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:57,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:57,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:57,977 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:57,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:57,976 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:57,977 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:57,977 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:57,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:57,977 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:57,977 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,077 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:58,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:58,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:58,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:58,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:58,081 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:58,081 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,081 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:58,081 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:58,081 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,081 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,084 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,084 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,084 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,084 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:58,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:58,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:58,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:58,085 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,085 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:58,085 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:58,086 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,088 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:58,089 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ded41[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:58,089 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,090 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,091 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:58,091 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:58,091 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,091 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:58,091 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:58,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:58,095 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:58,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:58,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:58,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:58,096 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:58,098 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,098 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,098 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,099 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,100 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:58,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:58,102 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,103 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,103 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,103 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,103 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:58,103 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,103 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:58,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507758160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:58,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507758166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,203 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:58,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:58,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:58,207 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:58,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:58,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:58,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,209 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:58,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:58,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:58,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:58,217 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,218 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:58,218 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:58,218 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:58,223 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:58,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ac49149[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:58,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:58,227 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,228 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:58,229 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,229 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:58,230 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:58,233 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:58,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:58,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:58,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:58,233 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:58,234 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,234 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,234 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,235 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,237 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:58,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,240 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,240 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:58,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,240 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:58,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/cf8e8bf2cc084a1d981e1824a81aa95f 2023-03-22 17:54:58,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/5a3dcae63d524c7a96f81ebf8ef69538 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5a3dcae63d524c7a96f81ebf8ef69538 2023-03-22 17:54:58,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5a3dcae63d524c7a96f81ebf8ef69538, entries=150, sequenceid=230, filesize=11.8 K 2023-03-22 17:54:58,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/2d6f14bca789452db9ca49d6724fe1dc as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/2d6f14bca789452db9ca49d6724fe1dc 2023-03-22 17:54:58,341 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:58,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:58,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:58,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:58,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:58,345 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:58,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:58,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:58,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/2d6f14bca789452db9ca49d6724fe1dc, entries=150, sequenceid=230, filesize=11.8 K 2023-03-22 17:54:58,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/cf8e8bf2cc084a1d981e1824a81aa95f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/cf8e8bf2cc084a1d981e1824a81aa95f 2023-03-22 17:54:58,346 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,347 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,347 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:58,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:58,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:58,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:58,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:58,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/cf8e8bf2cc084a1d981e1824a81aa95f, entries=150, sequenceid=230, filesize=11.8 K 2023-03-22 17:54:58,349 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,351 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:58,351 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:58,351 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:58,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:58,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:58,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:58,351 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a0a59fb2216419102dce85f7adab989d in 858ms, sequenceid=230, compaction requested=true 2023-03-22 17:54:58,351 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:58,351 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:58,351 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:58,351 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:58,351 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-22 17:54:58,352 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:58,353 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:58,353 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:54:58,353 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,353 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/7cc6441ac53548528f3c2f1c303bd750, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/f9d8cbeb899d43628159b79d0dcd807d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/2d6f14bca789452db9ca49d6724fe1dc] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=35.7 K 2023-03-22 17:54:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,354 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 7cc6441ac53548528f3c2f1c303bd750, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679507695676 2023-03-22 17:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:58,355 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f9d8cbeb899d43628159b79d0dcd807d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679507695763 2023-03-22 17:54:58,355 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2d6f14bca789452db9ca49d6724fe1dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679507697175 2023-03-22 17:54:58,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,357 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:58,357 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:54:58,373 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,373 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e64833f523064b13a6364b204f6ec0f3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/35fca7f94e8e407da4720cb2def34afd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5a3dcae63d524c7a96f81ebf8ef69538] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=35.7 K 2023-03-22 17:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,374 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e64833f523064b13a6364b204f6ec0f3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679507695676 2023-03-22 17:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,375 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 35fca7f94e8e407da4720cb2def34afd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679507695763 2023-03-22 17:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:58,375 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 5a3dcae63d524c7a96f81ebf8ef69538, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679507697175 2023-03-22 17:54:58,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:58,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fade300[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:58,376 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:58,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:58,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:58,377 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,377 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:58,380 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:58,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:58,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:58,382 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,387 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#247 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:58,388 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:58,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:58,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:58,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:58,388 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:58,390 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,390 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,390 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,391 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:58,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:58,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,396 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:58,396 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:58,407 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/2228685e78c64a7db4bf9f2869641ca2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2228685e78c64a7db4bf9f2869641ca2 2023-03-22 17:54:58,412 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into 2228685e78c64a7db4bf9f2869641ca2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:58,412 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:58,412 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507698351; duration=0sec 2023-03-22 17:54:58,412 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:58,412 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:58,413 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:58,414 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:54:58,414 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,414 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/60e12e43cf1b40798631cfbc43fa8b9a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1def152e4b24e789cba417c08641d10, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/cf8e8bf2cc084a1d981e1824a81aa95f] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=35.7 K 2023-03-22 17:54:58,414 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 60e12e43cf1b40798631cfbc43fa8b9a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679507695676 2023-03-22 17:54:58,415 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b1def152e4b24e789cba417c08641d10, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679507695763 2023-03-22 17:54:58,415 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting cf8e8bf2cc084a1d981e1824a81aa95f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679507697175 2023-03-22 17:54:58,421 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#248 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:58,434 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/511169be58634b91940333a1b8ab2bd0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/511169be58634b91940333a1b8ab2bd0 2023-03-22 17:54:58,438 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into 511169be58634b91940333a1b8ab2bd0(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:58,438 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:58,438 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507698351; duration=0sec 2023-03-22 17:54:58,438 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:58,496 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:58,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:58,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:58,499 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:58,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:58,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:58,500 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,500 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,501 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:58,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:58,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:58,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:58,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:58,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:58,503 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-22 17:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:58,504 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:58,504 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:58,504 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:54:58,512 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/ab6861be02374de18383ace2e9c95178 2023-03-22 17:54:58,525 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/6507a34877004646abafca99e4000bde 2023-03-22 17:54:58,534 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/430931344907493cb06767c3d39fa69e 2023-03-22 17:54:58,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/ab6861be02374de18383ace2e9c95178 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/ab6861be02374de18383ace2e9c95178 2023-03-22 17:54:58,543 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/ab6861be02374de18383ace2e9c95178, entries=150, sequenceid=267, filesize=11.9 K 2023-03-22 17:54:58,544 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/6507a34877004646abafca99e4000bde as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/6507a34877004646abafca99e4000bde 2023-03-22 17:54:58,547 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/6507a34877004646abafca99e4000bde, entries=150, sequenceid=267, filesize=11.9 K 2023-03-22 17:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/430931344907493cb06767c3d39fa69e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430931344907493cb06767c3d39fa69e 2023-03-22 17:54:58,552 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430931344907493cb06767c3d39fa69e, entries=150, sequenceid=267, filesize=11.9 K 2023-03-22 17:54:58,552 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for a0a59fb2216419102dce85f7adab989d in 49ms, sequenceid=267, compaction requested=false 2023-03-22 17:54:58,552 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:58,552 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:58,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:58,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:58,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:58,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:58,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd95f27[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:58,557 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:58,557 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,558 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,558 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:58,559 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,559 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,561 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,561 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:58,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:58,564 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:58,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:58,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:58,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:58,564 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:58,568 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,568 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,568 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,569 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,574 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,574 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,574 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,574 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,574 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,574 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:58,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:58,574 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,574 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,575 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:58,575 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:58,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:58,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:54:58,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:54:58,675 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:58,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:58,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:54:58,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:58,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:54:58,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:58,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:54:58,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:54:58,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:58,678 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:58,678 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:58,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:58,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:58,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:58,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:58,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:58,682 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,682 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:58,682 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:58,682 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:58,684 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:58,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c35278[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:58,686 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:58,686 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:58,687 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:58,687 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:58,692 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:58,692 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,692 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:58,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:58,695 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:58,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:58,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:58,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:58,695 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:58,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,697 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,698 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,701 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:58,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,702 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:58,701 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,702 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:58,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:58,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507698708, max timestamp=1679507698717Num uniques 859; 2023-03-22 17:54:58,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507698708, max timestamp=1679507698717Num uniques 859; 2023-03-22 17:54:58,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507698708, max timestamp=1679507698717Num uniques 859; 2023-03-22 17:54:58,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:58,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:58,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:54:58,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:58,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507758760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:58,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507758761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,793 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1f0236a618664613a3ce45f3038703ad as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f0236a618664613a3ce45f3038703ad 2023-03-22 17:54:58,797 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into 1f0236a618664613a3ce45f3038703ad(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:54:58,798 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:58,798 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507698351; duration=0sec 2023-03-22 17:54:58,798 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:58,802 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:58,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:58,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:58,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:58,805 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:58,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:58,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:58,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,807 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,808 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:58,809 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:58,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:58,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:58,824 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:58,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:58,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c73c7b0[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:58,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:58,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,832 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:58,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:58,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:58,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:58,838 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:58,838 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:58,838 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:58,838 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:58,838 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:58,840 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,840 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,840 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,840 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,842 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:58,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:58,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,847 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:58,851 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:58,851 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:58,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:58,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507758865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:58,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507758865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,951 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:58,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:58,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:58,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:58,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:58,954 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:58,954 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,954 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:58,954 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:58,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,956 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:58,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:58,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:58,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:58,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:58,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:58,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c468cdd[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:58,961 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:58,961 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,962 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:58,962 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,962 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:58,964 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:58,966 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:58,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:58,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:58,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:58,967 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:58,968 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,968 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,968 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:58,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:58,969 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,970 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:58,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:58,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:58,972 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:58,972 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:58,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:58,972 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:58,972 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:58,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507759070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,072 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:59,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507759071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:59,076 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:59,076 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,076 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:59,076 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:59,076 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,076 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,078 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,078 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:59,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:59,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:59,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:59,079 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,079 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:59,079 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:59,079 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:59,082 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,083 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:59,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5953c615[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:59,084 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:59,084 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,085 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,085 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,085 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,086 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,088 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/a22d15f906e544b19c94af1078b7e975 2023-03-22 17:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:59,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:59,090 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:59,090 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:59,090 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:59,090 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:59,091 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:59,097 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,097 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,097 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,098 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,100 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,102 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:59,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,103 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:59,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:59,103 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:59,203 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:59,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:59,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:59,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:59,206 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:59,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:59,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:59,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,208 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:59,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:59,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:59,210 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,211 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:59,211 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:59,211 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,212 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:59,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510b0fd5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:59,214 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:59,214 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,216 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:59,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:59,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:59,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:59,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:59,220 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:59,220 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:59,220 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:59,220 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:59,220 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:59,222 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,222 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,222 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,224 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:59,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:59,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,227 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:59,227 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:59,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:59,327 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:59,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:59,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:59,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:59,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:59,331 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:59,331 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,331 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:59,331 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:59,332 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,332 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,334 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:59,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:59,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:59,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:59,335 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:59,335 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,335 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:59,335 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:59,336 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,336 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:59,336 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:59,336 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:59,337 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,339 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,339 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:59,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23370c03[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:59,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:59,341 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,342 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,342 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:59,342 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:59,342 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:59,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:59,343 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:59,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:59,346 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:59,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:59,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:59,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:59,346 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:59,349 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,349 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,349 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,355 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:59,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:59,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,355 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:59,355 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:59,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:59,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507759380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507759380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,455 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:59,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:59,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:59,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:59,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:59,459 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:59,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:59,459 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:59,460 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,460 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,461 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,461 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,461 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,461 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:59,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:59,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:59,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:59,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:59,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:59,463 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,463 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:59,463 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:59,463 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:59,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:59,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:59,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b4f4cdc[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:59,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,468 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:59,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:59,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:59,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:59,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:59,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:59,473 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:59,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:59,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:59,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:59,473 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,477 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,483 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:59,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:59,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,484 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,484 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,484 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:59,484 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,484 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,484 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/774cc840e66c4926b3c3dcfbb341a32a 2023-03-22 17:54:59,584 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:59,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:59,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:59,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:59,588 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:59,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:59,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:59,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,589 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,589 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,590 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,590 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:59,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:59,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:59,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:59,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:59,591 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:59,592 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,592 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:59,592 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:59,592 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,594 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:59,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ceb4ece[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:59,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:59,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,599 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:59,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:59,616 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:59,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:59,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:59,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:59,616 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:59,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,628 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:59,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:59,629 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:59,630 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,730 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:59,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:59,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:59,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:59,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:59,736 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:59,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:59,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:59,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,738 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:59,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:59,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:59,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:59,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:59,748 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,748 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:59,748 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:59,748 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:59,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:59,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35978b21[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:59,754 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:59,754 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,757 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:59,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:59,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:59,758 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:59,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:59,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:59,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:59,758 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:59,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,762 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,778 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:59,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:59,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,779 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:59,779 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:59,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,879 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:54:59,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:54:59,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:54:59,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:54:59,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:54:59,882 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:54:59,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:54:59,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:54:59,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,884 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:54:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:54:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:54:59,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:54:59,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:54:59,887 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,887 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:54:59,887 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:54:59,887 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:54:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:54:59,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:59,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507759887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,889 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,889 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,890 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,890 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:54:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507759888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:54:59,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e5bde84[Count = 0] remaining members to acquire global barrier 2023-03-22 17:54:59,891 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:54:59,891 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:54:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:54:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:54:59,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:54:59,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:54:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:54:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:54:59,897 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:54:59,898 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:54:59,898 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:54:59,898 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:54:59,898 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:54:59,899 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,899 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,899 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:54:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:54:59,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,900 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,901 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:54:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:54:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:54:59,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:54:59,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:54:59,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:54:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:54:59,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:54:59,904 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:54:59,904 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:54:59,904 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:54:59,904 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:54:59,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:54:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:54:59,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507759908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,911 DEBUG [Thread-1637] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:59,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/e428b7957912434096ff5c0d2364ee43 2023-03-22 17:54:59,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/a22d15f906e544b19c94af1078b7e975 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/a22d15f906e544b19c94af1078b7e975 2023-03-22 17:54:59,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:54:59,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54128 deadline: 1679507759919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:54:59,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/a22d15f906e544b19c94af1078b7e975, entries=200, sequenceid=278, filesize=14.3 K 2023-03-22 17:54:59,920 DEBUG [Thread-1635] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:54:59,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/774cc840e66c4926b3c3dcfbb341a32a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/774cc840e66c4926b3c3dcfbb341a32a 2023-03-22 17:54:59,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/774cc840e66c4926b3c3dcfbb341a32a, entries=150, sequenceid=278, filesize=11.9 K 2023-03-22 17:54:59,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/e428b7957912434096ff5c0d2364ee43 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e428b7957912434096ff5c0d2364ee43 2023-03-22 17:54:59,928 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e428b7957912434096ff5c0d2364ee43, entries=150, sequenceid=278, filesize=11.9 K 2023-03-22 17:54:59,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a0a59fb2216419102dce85f7adab989d in 1255ms, sequenceid=278, compaction requested=true 2023-03-22 17:54:59,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:54:59,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:54:59,929 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:59,929 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:54:59,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:59,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:54:59,930 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39394 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:59,930 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:54:59,930 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:54:59,930 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:54:59,930 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,930 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:54:59,930 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2228685e78c64a7db4bf9f2869641ca2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/ab6861be02374de18383ace2e9c95178, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/a22d15f906e544b19c94af1078b7e975] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=38.5 K 2023-03-22 17:54:59,930 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f0236a618664613a3ce45f3038703ad, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/6507a34877004646abafca99e4000bde, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/774cc840e66c4926b3c3dcfbb341a32a] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.1 K 2023-03-22 17:54:59,931 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2228685e78c64a7db4bf9f2869641ca2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679507697175 2023-03-22 17:54:59,931 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1f0236a618664613a3ce45f3038703ad, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679507697175 2023-03-22 17:54:59,931 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting ab6861be02374de18383ace2e9c95178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679507697545 2023-03-22 17:54:59,931 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6507a34877004646abafca99e4000bde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679507697545 2023-03-22 17:54:59,932 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a22d15f906e544b19c94af1078b7e975, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679507698668 2023-03-22 17:54:59,932 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 774cc840e66c4926b3c3dcfbb341a32a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679507698668 2023-03-22 17:54:59,938 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:54:59,938 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#256 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:00,005 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:00,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:00,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:00,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:00,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:00,008 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:00,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:00,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:00,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,013 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:00,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:00,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:00,014 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:00,014 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,014 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:00,014 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:00,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:00,015 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-22 17:55:00,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:00,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:00,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:00,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:00,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:00,015 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:00,024 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/2dbb26d183434e98a8fe5bc7092a5300 2023-03-22 17:55:00,035 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/edf60a71058b410d82e79970214ac543 2023-03-22 17:55:00,350 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/dec1ada75ff2490f8cde6f4252114b01 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/dec1ada75ff2490f8cde6f4252114b01 2023-03-22 17:55:00,355 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into dec1ada75ff2490f8cde6f4252114b01(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:00,355 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:00,355 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507699929; duration=0sec 2023-03-22 17:55:00,355 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:00,355 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:00,356 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:00,356 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:55:00,356 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,356 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/511169be58634b91940333a1b8ab2bd0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430931344907493cb06767c3d39fa69e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e428b7957912434096ff5c0d2364ee43] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.1 K 2023-03-22 17:55:00,357 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 511169be58634b91940333a1b8ab2bd0, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679507697175 2023-03-22 17:55:00,357 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/cdea144660e04c66bb9edb2118e666e7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/cdea144660e04c66bb9edb2118e666e7 2023-03-22 17:55:00,357 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 430931344907493cb06767c3d39fa69e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679507697545 2023-03-22 17:55:00,357 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e428b7957912434096ff5c0d2364ee43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679507698668 2023-03-22 17:55:00,363 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into cdea144660e04c66bb9edb2118e666e7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:00,363 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:00,363 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507699929; duration=0sec 2023-03-22 17:55:00,363 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:00,366 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#260 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:00,375 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/d1a8fe654044428b870c5e566bed7bce as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d1a8fe654044428b870c5e566bed7bce 2023-03-22 17:55:00,379 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into d1a8fe654044428b870c5e566bed7bce(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:00,379 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:00,379 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507699929; duration=0sec 2023-03-22 17:55:00,379 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:00,446 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/50d2e3cf08354ccb89ec74cb14415a96 2023-03-22 17:55:00,450 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/2dbb26d183434e98a8fe5bc7092a5300 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2dbb26d183434e98a8fe5bc7092a5300 2023-03-22 17:55:00,453 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2dbb26d183434e98a8fe5bc7092a5300, entries=150, sequenceid=314, filesize=11.9 K 2023-03-22 17:55:00,454 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/edf60a71058b410d82e79970214ac543 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/edf60a71058b410d82e79970214ac543 2023-03-22 17:55:00,460 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/edf60a71058b410d82e79970214ac543, entries=150, sequenceid=314, filesize=11.9 K 2023-03-22 17:55:00,461 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/50d2e3cf08354ccb89ec74cb14415a96 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/50d2e3cf08354ccb89ec74cb14415a96 2023-03-22 17:55:00,464 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/50d2e3cf08354ccb89ec74cb14415a96, entries=150, sequenceid=314, filesize=11.9 K 2023-03-22 17:55:00,465 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for a0a59fb2216419102dce85f7adab989d in 450ms, sequenceid=314, compaction requested=false 2023-03-22 17:55:00,465 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:00,465 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:00,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:00,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f735e3f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:00,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:00,469 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,470 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:00,471 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,471 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:00,472 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,472 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,472 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:00,472 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:00,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:00,475 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:00,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:00,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:00,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:00,475 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:00,478 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,478 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:00,478 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,478 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,479 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,482 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,482 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:00,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:00,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:00,482 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,483 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:00,483 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:00,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:00,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:00,483 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:00,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,483 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,583 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:00,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:00,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:00,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:00,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:00,586 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:00,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:00,586 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:00,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,588 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:00,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:00,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:00,590 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,590 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:00,590 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:00,590 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:00,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:00,592 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:00,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4908f821[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:00,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:00,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:00,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:00,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:00,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:00,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:00,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:00,600 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:00,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:00,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:00,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:00,600 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:00,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:00,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:00,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:00,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:00,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,607 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:00,607 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:00,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:00,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:00,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,707 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:00,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:00,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:00,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:00,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:00,711 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:00,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:00,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:00,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,713 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:00,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:00,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:00,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:00,715 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,715 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:00,716 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:00,716 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:00,724 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40cfaca2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:00,726 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:00,726 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,728 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:00,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:00,730 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:00,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:00,736 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:00,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:00,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:00,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:00,736 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:00,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:00,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:00,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:00,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:00,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:00,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,744 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:00,744 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:00,744 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:00,744 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,744 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,744 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,744 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,844 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:00,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:00,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:00,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:00,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:00,848 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:00,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:00,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:00,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:00,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:00,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:00,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:00,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:00,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:00,875 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:00,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3090e5c1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:00,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:00,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:00,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:00,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:00,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:00,882 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:00,882 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:00,882 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:00,882 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:00,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:00,885 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:00,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:00,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:00,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:00,885 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:00,888 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,888 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:00,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,888 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:00,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,889 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:00,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:00,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:00,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:00,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:00,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,894 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,894 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,894 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,894 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:00,894 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:00,894 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:00,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:00,894 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:00,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:00,904 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:00,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:00,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507700931, max timestamp=1679507700938Num uniques 859; 2023-03-22 17:55:00,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507700931, max timestamp=1679507700938Num uniques 859; 2023-03-22 17:55:00,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:00,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:00,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507700931, max timestamp=1679507700938Num uniques 859; 2023-03-22 17:55:00,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:00,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:00,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507760964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:00,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507760967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,994 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:00,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:00,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:00,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:00,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:00,997 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:00,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:00,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:00,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:00,998 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:00,998 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,999 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:00,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:00,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:00,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:01,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:01,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:01,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:01,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:01,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:01,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:01,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:01,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:01,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:01,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:01,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:01,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:01,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:01,003 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:01,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a7c74a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:01,004 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:01,004 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,005 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,005 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:01,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:01,008 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:01,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,008 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,008 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:01,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:01,010 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:01,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:01,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:01,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:01,011 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:01,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,015 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,017 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:01,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:01,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,017 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:01,017 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:01,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:01,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507761068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:01,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507761071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,117 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:01,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:01,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:01,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:01,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:01,120 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:01,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:01,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:01,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,122 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,122 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,122 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,122 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:01,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:01,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:01,124 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,124 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:01,124 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:01,124 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:01,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@be0ee2f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:01,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:01,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:01,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:01,132 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:01,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:01,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:01,135 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:01,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:01,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:01,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:01,135 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:01,136 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,136 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,136 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,138 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,141 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,141 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,141 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,141 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:01,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:01,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,141 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,141 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,141 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,141 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:01,141 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:01,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:01,141 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,142 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,142 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,241 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:01,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:01,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:01,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:01,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:01,245 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:01,245 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,245 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:01,245 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:01,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,248 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,248 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,248 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:01,250 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,250 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:01,250 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:01,250 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f7f94d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:01,254 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:01,254 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,257 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,257 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:01,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:01,263 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:01,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:01,267 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:01,267 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:01,267 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:01,267 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:01,267 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:01,268 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,269 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,269 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:01,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:01,269 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:01,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507761270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,276 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,276 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:01,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:01,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,276 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:01,276 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:01,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:01,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:01,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507761275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/b896f3c533974e1d8be3e6925e4b15e1 2023-03-22 17:55:01,376 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:01,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:01,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:01,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:01,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:01,379 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:01,379 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,380 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:01,380 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:01,380 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,380 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,381 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,381 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,382 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,382 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:01,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:01,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:01,383 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,383 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:01,383 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:01,383 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:01,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1febcbf[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:01,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:01,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,389 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:01,389 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,389 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:01,391 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:01,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,391 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,391 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:01,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:01,393 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:01,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:01,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:01,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:01,393 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:01,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,399 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,399 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,400 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,400 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:01,400 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:01,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:01,500 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:01,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:01,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:01,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:01,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:01,504 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:01,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:01,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:01,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:01,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:01,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:01,507 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,507 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:01,508 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:01,508 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:01,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:01,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:01,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:01,512 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:01,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ad8b65d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:01,514 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:01,514 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:01,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:01,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:01,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:01,517 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,517 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:01,517 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,517 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:01,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:01,520 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:01,520 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:01,520 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:01,520 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:01,520 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:01,521 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,521 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,521 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,523 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,526 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:01,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,526 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:01,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,526 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:01,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:01,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54132 deadline: 1679507761575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:01,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54212 deadline: 1679507761580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,626 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:01,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:01,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:01,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:01,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:01,630 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:01,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:01,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:01,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,631 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:01,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:01,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:01,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:01,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:01,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:01,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:01,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3645a6bf[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:01,638 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:01,638 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:01,640 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,640 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:01,641 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:01,641 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,641 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,641 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:01,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:01,643 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:01,643 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:01,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:01,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:01,644 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:01,649 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,649 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,649 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,650 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,651 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,654 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,654 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:01,654 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:01,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,654 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,655 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:01,655 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:01,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1bb3062e455f43d1b783d4855e697e67 2023-03-22 17:55:01,755 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:01,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:01,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:01,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:01,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:01,758 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:01,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:01,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:01,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,760 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,760 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,760 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:01,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:01,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:01,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:01,761 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:01,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:01,762 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:01,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:01,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:01,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f711cab[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:01,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:01,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:01,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,769 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:01,769 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,769 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:01,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:01,772 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:01,772 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:01,772 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:01,772 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:01,772 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:01,773 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,773 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,774 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,774 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,775 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:01,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:01,778 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:01,779 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:01,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,779 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:01,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,879 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:01,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:01,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:01,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:01,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:01,882 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:01,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:01,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:01,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,885 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:01,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:01,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:01,886 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,886 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:01,886 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:01,886 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:01,888 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:01,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9e54be[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:01,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:01,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,892 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:01,892 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,892 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:01,892 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:01,892 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:01,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:01,896 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:01,896 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:01,896 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:01,896 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:01,896 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:01,898 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,898 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:01,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,898 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:01,899 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,900 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:01,902 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:01,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:01,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:01,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:01,902 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:01,903 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:01,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:01,903 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:02,003 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:02,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:02,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:02,006 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:02,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:02,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:02,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:02,009 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:02,009 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,009 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:02,010 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,010 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:02,010 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:02,010 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:02,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,012 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,012 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:02,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@348b69ce[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:02,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:02,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:02,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:02,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:02,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:02,020 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:02,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:02,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:02,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:02,020 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:02,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,022 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,025 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:02,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:02,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,031 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:02,031 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:02,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54132 deadline: 1679507762079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:02,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507762082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,131 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:02,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:02,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:02,134 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:02,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:02,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:02,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,136 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:02,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:02,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:02,138 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,138 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:02,138 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:02,138 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:02,139 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/5bafa8daac6043e3a162adcaf495eae2 2023-03-22 17:55:02,140 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,141 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,141 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:02,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f6041be[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:02,143 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:02,143 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:02,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:02,144 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/b896f3c533974e1d8be3e6925e4b15e1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/b896f3c533974e1d8be3e6925e4b15e1 2023-03-22 17:55:02,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:02,144 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,145 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:02,147 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,148 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,148 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/b896f3c533974e1d8be3e6925e4b15e1, entries=150, sequenceid=331, filesize=11.9 K 2023-03-22 17:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1bb3062e455f43d1b783d4855e697e67 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1bb3062e455f43d1b783d4855e697e67 2023-03-22 17:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:02,150 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:02,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:02,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:02,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:02,151 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:02,152 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,152 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,152 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,153 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1bb3062e455f43d1b783d4855e697e67, entries=150, sequenceid=331, filesize=11.9 K 2023-03-22 17:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/5bafa8daac6043e3a162adcaf495eae2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/5bafa8daac6043e3a162adcaf495eae2 2023-03-22 17:55:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/5bafa8daac6043e3a162adcaf495eae2, entries=150, sequenceid=331, filesize=11.9 K 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,157 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:02,157 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a0a59fb2216419102dce85f7adab989d in 1253ms, sequenceid=331, compaction requested=true 2023-03-22 17:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:02,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:02,157 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,158 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:02,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:02,158 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:02,158 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:02,158 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:02,158 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:02,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:02,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:02,160 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:02,160 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:02,160 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:55:02,160 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:55:02,160 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,160 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,160 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/dec1ada75ff2490f8cde6f4252114b01, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/edf60a71058b410d82e79970214ac543, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1bb3062e455f43d1b783d4855e697e67] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.3 K 2023-03-22 17:55:02,160 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/cdea144660e04c66bb9edb2118e666e7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2dbb26d183434e98a8fe5bc7092a5300, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/b896f3c533974e1d8be3e6925e4b15e1] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.3 K 2023-03-22 17:55:02,160 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting dec1ada75ff2490f8cde6f4252114b01, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679507698668 2023-03-22 17:55:02,161 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting cdea144660e04c66bb9edb2118e666e7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679507698668 2023-03-22 17:55:02,161 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting edf60a71058b410d82e79970214ac543, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679507698732 2023-03-22 17:55:02,161 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2dbb26d183434e98a8fe5bc7092a5300, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679507698732 2023-03-22 17:55:02,161 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1bb3062e455f43d1b783d4855e697e67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679507700900 2023-03-22 17:55:02,162 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b896f3c533974e1d8be3e6925e4b15e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679507700900 2023-03-22 17:55:02,169 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:02,169 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:02,258 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:02,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:02,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:02,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:02,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:02,261 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:02,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:02,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:02,262 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,262 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,263 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,263 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,263 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,263 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:02,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:02,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:02,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:02,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:02,265 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,265 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:02,265 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-22 17:55:02,265 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:02,265 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:02,265 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:02,265 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:02,265 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:02,265 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:02,580 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/451ccf1560a54ad29dbef175bfe81f9e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/451ccf1560a54ad29dbef175bfe81f9e 2023-03-22 17:55:02,581 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/89b6b61299e24711a9cceff136cfa8fe as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/89b6b61299e24711a9cceff136cfa8fe 2023-03-22 17:55:02,596 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into 451ccf1560a54ad29dbef175bfe81f9e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:02,596 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:02,596 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507702158; duration=0sec 2023-03-22 17:55:02,596 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:02,596 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:02,598 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:02,598 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into 89b6b61299e24711a9cceff136cfa8fe(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:02,598 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:02,598 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507702158; duration=0sec 2023-03-22 17:55:02,598 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:02,598 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:55:02,598 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,599 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d1a8fe654044428b870c5e566bed7bce, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/50d2e3cf08354ccb89ec74cb14415a96, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/5bafa8daac6043e3a162adcaf495eae2] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.3 K 2023-03-22 17:55:02,599 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d1a8fe654044428b870c5e566bed7bce, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679507698668 2023-03-22 17:55:02,599 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 50d2e3cf08354ccb89ec74cb14415a96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679507698732 2023-03-22 17:55:02,600 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5bafa8daac6043e3a162adcaf495eae2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679507700900 2023-03-22 17:55:02,620 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#267 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:02,672 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/2c3095626b984f59abb1ddfd1d05c471 2023-03-22 17:55:02,682 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/00bf964e611d446e86ed069abe4b9879 2023-03-22 17:55:02,691 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/aba3726d3e794e9eac68b5532fa05e94 2023-03-22 17:55:02,695 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/2c3095626b984f59abb1ddfd1d05c471 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2c3095626b984f59abb1ddfd1d05c471 2023-03-22 17:55:02,698 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2c3095626b984f59abb1ddfd1d05c471, entries=150, sequenceid=366, filesize=11.9 K 2023-03-22 17:55:02,699 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/00bf964e611d446e86ed069abe4b9879 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/00bf964e611d446e86ed069abe4b9879 2023-03-22 17:55:02,702 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/00bf964e611d446e86ed069abe4b9879, entries=150, sequenceid=366, filesize=11.9 K 2023-03-22 17:55:02,703 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/aba3726d3e794e9eac68b5532fa05e94 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aba3726d3e794e9eac68b5532fa05e94 2023-03-22 17:55:02,706 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aba3726d3e794e9eac68b5532fa05e94, entries=150, sequenceid=366, filesize=11.9 K 2023-03-22 17:55:02,707 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for a0a59fb2216419102dce85f7adab989d in 442ms, sequenceid=366, compaction requested=false 2023-03-22 17:55:02,707 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:02,707 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:02,709 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18c384bd[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:02,712 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:02,712 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,713 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:02,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,713 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:02,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:02,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:02,715 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:02,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:02,718 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:02,718 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:02,718 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:02,718 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:02,718 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:02,719 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,719 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,719 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,725 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,725 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:02,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:02,725 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,725 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,726 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:02,726 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:02,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,826 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:02,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:02,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:02,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:02,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:02,829 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:02,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:02,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:02,829 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:02,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:02,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:02,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:02,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:02,834 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,834 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:02,834 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:02,834 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:02,845 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:02,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@214108bf[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:02,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:02,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,848 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:02,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:02,849 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:02,849 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:02,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:02,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:02,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:02,853 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:02,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:02,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:02,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:02,854 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:02,855 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,855 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,855 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,856 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,858 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,863 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,863 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,863 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,863 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:02,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:02,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,864 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:02,864 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:02,863 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,863 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,964 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:02,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:02,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:02,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:02,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:02,968 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:02,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:02,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:02,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,970 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,970 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:02,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:02,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:02,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:02,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:02,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:02,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:02,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:02,973 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,973 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:02,973 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:02,974 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:02,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:02,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:02,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:02,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:02,975 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,975 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,976 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:02,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eebd040[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:02,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:02,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,979 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:02,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:02,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:02,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:02,981 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:02,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:02,983 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:02,983 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:02,983 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:02,984 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:02,984 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:02,986 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,986 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,986 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:02,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:02,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:02,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:02,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:02,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:02,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:02,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:02,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:02,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:02,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:02,991 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:02,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:02,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:02,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:02,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:02,991 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:02,991 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:03,030 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/f1b2d17d931f4a588ad67bd3b6de0099 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f1b2d17d931f4a588ad67bd3b6de0099 2023-03-22 17:55:03,034 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into f1b2d17d931f4a588ad67bd3b6de0099(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:03,034 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:03,035 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507702158; duration=0sec 2023-03-22 17:55:03,035 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:03,091 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:03,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:03,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:03,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:03,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:03,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:03,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:03,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:03,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:03,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:03,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:03,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:03,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:03,095 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:03,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:03,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:03,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,096 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,097 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:03,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:03,099 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,099 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:03,099 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:03,099 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:03,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:03,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:03,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:03,101 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,101 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,102 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,102 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@632b0592[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:03,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:03,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/5413be8516534cd0804ad8612356306e 2023-03-22 17:55:03,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,107 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:03,107 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:03,107 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:03,107 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,107 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:03,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:03,111 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:03,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:03,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:03,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:03,112 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:03,114 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,114 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,114 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,115 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,116 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:03,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,119 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:03,120 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:03,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507763123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507703100, max timestamp=1679507703124Num uniques 859; 2023-03-22 17:55:03,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:03,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507703100, max timestamp=1679507703124Num uniques 859; 2023-03-22 17:55:03,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:03,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507703100, max timestamp=1679507703124Num uniques 859; 2023-03-22 17:55:03,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:03,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54132 deadline: 1679507763172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,220 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:03,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:03,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:03,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:03,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:03,223 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:03,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:03,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:03,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,225 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:03,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:03,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:03,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:03,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:03,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:03,226 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:03,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:03,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:03,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507763226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,228 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67499326[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:03,230 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:03,230 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,231 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:03,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:03,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:03,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:03,232 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:03,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:03,234 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:03,234 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:03,234 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:03,234 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:03,234 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:03,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,238 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,239 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:03,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:03,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:03,241 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:03,241 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:03,241 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54132 deadline: 1679507763276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,342 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:03,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:03,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:03,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:03,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:03,345 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:03,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:03,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:03,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,347 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,347 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,347 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:03,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:03,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:03,349 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,349 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:03,349 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:03,349 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e646b80[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:03,357 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:03,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:03,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:03,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:03,363 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:03,363 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:03,363 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:03,363 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:03,363 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:03,365 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,365 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,365 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:03,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:03,372 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,372 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:03,372 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507763429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,473 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:03,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:03,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:03,476 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:03,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:03,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:03,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,477 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,477 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,478 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,478 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:03,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:03,479 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,479 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:03,479 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:03,480 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,480 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:03,480 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:03,480 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,480 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:03,480 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:03,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,481 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:03,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1ddbdc[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:03,483 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:03,483 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54132 deadline: 1679507763482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,485 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:03,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:03,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:03,486 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,487 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:03,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:03,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:03,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:03,490 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:03,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:03,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:03,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:03,490 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:03,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,496 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:03,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,496 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:03,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,496 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:03,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/4c803e0d45474852ba0a6e92aad4ea65 2023-03-22 17:55:03,597 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:03,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:03,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:03,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:03,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:03,600 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:03,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:03,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:03,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:03,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:03,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:03,605 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:03,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65a14218[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:03,609 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:03,609 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,610 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:03,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:03,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:03,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:03,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:03,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:03,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:03,615 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:03,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:03,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:03,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:03,615 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:03,617 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:03,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:03,623 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,623 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:03,623 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,723 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:03,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:03,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:03,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:03,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:03,727 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:03,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:03,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:03,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,728 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:03,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:03,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:03,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:03,731 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,731 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:03,731 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:03,731 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:03,733 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:03,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507763734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:03,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e5a2d29[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:03,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:03,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:03,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:03,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:03,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:03,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:03,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:03,743 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:03,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:03,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:03,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:03,743 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:03,746 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,747 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,747 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,747 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:03,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:03,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,752 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,752 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,752 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,752 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,752 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:03,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:03,752 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:03,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54132 deadline: 1679507763785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,852 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:03,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:03,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:03,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:03,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:03,855 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:03,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:03,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:03,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,856 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,857 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,857 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,857 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,857 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:03,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:03,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:03,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:03,860 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,860 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:03,860 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:03,860 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:03,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@230f6709[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:03,865 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:03,865 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,866 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:03,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:03,869 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,869 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,869 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,869 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:03,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:03,871 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:03,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:03,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:03,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:03,871 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:03,873 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,873 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,873 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:03,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:03,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:03,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:03,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,879 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:03,879 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:03,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:03,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54128 deadline: 1679507763923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,925 DEBUG [Thread-1635] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:03,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/e7a92f9d904542f1ae55e92b73105e88 2023-03-22 17:55:03,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/5413be8516534cd0804ad8612356306e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5413be8516534cd0804ad8612356306e 2023-03-22 17:55:03,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:03,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54160 deadline: 1679507763935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,939 DEBUG [Thread-1637] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:03,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5413be8516534cd0804ad8612356306e, entries=150, sequenceid=382, filesize=11.9 K 2023-03-22 17:55:03,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/4c803e0d45474852ba0a6e92aad4ea65 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/4c803e0d45474852ba0a6e92aad4ea65 2023-03-22 17:55:03,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/4c803e0d45474852ba0a6e92aad4ea65, entries=100, sequenceid=382, filesize=9.5 K 2023-03-22 17:55:03,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/e7a92f9d904542f1ae55e92b73105e88 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e7a92f9d904542f1ae55e92b73105e88 2023-03-22 17:55:03,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e7a92f9d904542f1ae55e92b73105e88, entries=100, sequenceid=382, filesize=9.5 K 2023-03-22 17:55:03,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a0a59fb2216419102dce85f7adab989d in 857ms, sequenceid=382, compaction requested=true 2023-03-22 17:55:03,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:03,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:03,951 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:03,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:03,951 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:03,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:03,952 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37304 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:03,952 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:55:03,952 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,952 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/451ccf1560a54ad29dbef175bfe81f9e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2c3095626b984f59abb1ddfd1d05c471, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5413be8516534cd0804ad8612356306e] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.4 K 2023-03-22 17:55:03,952 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:03,952 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:55:03,953 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,953 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/89b6b61299e24711a9cceff136cfa8fe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/00bf964e611d446e86ed069abe4b9879, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/4c803e0d45474852ba0a6e92aad4ea65] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.0 K 2023-03-22 17:55:03,953 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 451ccf1560a54ad29dbef175bfe81f9e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679507700900 2023-03-22 17:55:03,953 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 89b6b61299e24711a9cceff136cfa8fe, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679507700900 2023-03-22 17:55:03,953 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2c3095626b984f59abb1ddfd1d05c471, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679507700955 2023-03-22 17:55:03,953 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 00bf964e611d446e86ed069abe4b9879, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679507700955 2023-03-22 17:55:03,953 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5413be8516534cd0804ad8612356306e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507703086 2023-03-22 17:55:03,954 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4c803e0d45474852ba0a6e92aad4ea65, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507703092 2023-03-22 17:55:03,959 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#273 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:03,959 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#274 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:03,970 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/a976ca1fbd7d4a4e85018991ac30d44c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/a976ca1fbd7d4a4e85018991ac30d44c 2023-03-22 17:55:03,971 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/3c55b3ca8d894d23a499e50ab5106ef1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/3c55b3ca8d894d23a499e50ab5106ef1 2023-03-22 17:55:03,974 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into a976ca1fbd7d4a4e85018991ac30d44c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:03,974 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:03,974 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507703951; duration=0sec 2023-03-22 17:55:03,974 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:03,974 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:03,975 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into 3c55b3ca8d894d23a499e50ab5106ef1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:03,975 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:03,975 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507703951; duration=0sec 2023-03-22 17:55:03,975 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:03,975 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:03,976 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:55:03,976 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,976 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f1b2d17d931f4a588ad67bd3b6de0099, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aba3726d3e794e9eac68b5532fa05e94, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e7a92f9d904542f1ae55e92b73105e88] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.0 K 2023-03-22 17:55:03,976 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f1b2d17d931f4a588ad67bd3b6de0099, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679507700900 2023-03-22 17:55:03,976 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting aba3726d3e794e9eac68b5532fa05e94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679507700955 2023-03-22 17:55:03,977 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e7a92f9d904542f1ae55e92b73105e88, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507703092 2023-03-22 17:55:03,979 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:03,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:03,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:03,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:03,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:03,982 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:03,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:03,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:03,983 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:03,983 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,983 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,985 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,985 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:03,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:03,985 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:03,985 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:03,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:03,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:03,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:03,988 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:03,988 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:03,989 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-22 17:55:03,989 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:03,989 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:03,989 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:03,989 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:03,989 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:03,989 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:04,002 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/a28911004b004395b5ad3eedfea272a9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/a28911004b004395b5ad3eedfea272a9 2023-03-22 17:55:04,005 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/1c0100ad68d04252b21704617c87cddb 2023-03-22 17:55:04,007 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into a28911004b004395b5ad3eedfea272a9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:04,007 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:04,007 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507703951; duration=0sec 2023-03-22 17:55:04,007 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:04,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:04,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:04,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:04,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507764265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:04,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54132 deadline: 1679507764289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:04,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507764366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,416 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/c0431b2046fd494d9a3628ca63055a50 2023-03-22 17:55:04,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:04,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507764569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,829 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/d8e2b0689c484bf0b7e9627f6cffd508 2023-03-22 17:55:04,833 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/1c0100ad68d04252b21704617c87cddb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/1c0100ad68d04252b21704617c87cddb 2023-03-22 17:55:04,837 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/1c0100ad68d04252b21704617c87cddb, entries=150, sequenceid=419, filesize=11.9 K 2023-03-22 17:55:04,838 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/c0431b2046fd494d9a3628ca63055a50 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c0431b2046fd494d9a3628ca63055a50 2023-03-22 17:55:04,841 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c0431b2046fd494d9a3628ca63055a50, entries=150, sequenceid=419, filesize=11.9 K 2023-03-22 17:55:04,842 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/d8e2b0689c484bf0b7e9627f6cffd508 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d8e2b0689c484bf0b7e9627f6cffd508 2023-03-22 17:55:04,846 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d8e2b0689c484bf0b7e9627f6cffd508, entries=150, sequenceid=419, filesize=11.9 K 2023-03-22 17:55:04,846 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for a0a59fb2216419102dce85f7adab989d in 858ms, sequenceid=419, compaction requested=false 2023-03-22 17:55:04,847 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:04,847 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:04,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:04,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:04,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:04,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:04,849 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:04,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:04,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:04,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137c8b8f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:04,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:04,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,854 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:04,854 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:04,854 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:04,855 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,855 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:04,855 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:04,855 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:04,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:04,858 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:04,858 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:04,858 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:04,858 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:04,858 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:04,860 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,860 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:04,860 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:04,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:04,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:04,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:04,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:04,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:04,865 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:04,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:04,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:04,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:04,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,866 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,866 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:04,866 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:04,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:04,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:04,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-22 17:55:04,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:04,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:04,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:04,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:04,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:04,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:04,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:04,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507764923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,966 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:04,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:04,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:04,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:04,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:04,969 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:04,969 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:04,969 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:04,969 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:04,969 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,969 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:04,970 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:04,970 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:04,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:04,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:04,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:04,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:04,972 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:04,972 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:04,972 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:04,972 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:04,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:04,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c22ba7a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:04,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:04,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,977 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,977 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:04,977 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:04,977 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:04,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:04,979 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:04,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:04,979 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:04,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:04,982 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:04,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:04,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:04,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:04,982 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:04,986 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,986 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:04,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,987 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:04,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:04,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:04,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:04,990 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:04,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:04,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:04,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:04,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:04,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,991 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:04,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:04,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:04,991 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:04,991 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:04,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:04,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:04,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:04,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:04,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:05,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507765027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,091 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:05,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:05,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:05,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:05,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:05,094 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:05,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:05,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:05,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,096 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,096 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,096 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,096 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:05,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:05,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:05,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:05,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:05,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:05,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:05,098 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,098 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:05,098 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:05,098 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:05,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,100 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:05,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@630ef3b2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:05,102 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:05,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,108 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,108 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:05,108 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:05,108 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:05,109 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:05,109 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,109 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:05,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:05,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:05,110 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:05,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:05,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:05,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:05,110 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:05,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,112 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,118 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,118 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,118 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,118 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,118 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:05,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:05,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:05,118 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,119 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:05,119 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:05,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,219 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:05,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:05,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:05,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:05,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:05,222 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:05,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:05,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:05,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,224 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,224 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,224 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,224 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:05,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:05,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:05,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:05,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:05,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:05,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:05,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:05,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:05,225 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,226 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:05,226 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:05,226 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:05,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:05,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be1f7f5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:05,230 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:05,230 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,231 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,231 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:05,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:05,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507765231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:05,234 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,234 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:05,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:05,237 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:05,237 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:05,237 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:05,237 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:05,237 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:05,239 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,239 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,239 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,240 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:05,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,244 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:05,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,244 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:05,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/e023a44e7c1b4ae7ad93074933f92f9e 2023-03-22 17:55:05,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:05,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54132 deadline: 1679507765296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,344 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:05,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:05,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:05,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:05,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:05,348 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:05,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:05,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:05,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:05,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:05,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:05,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:05,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:05,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:05,352 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,352 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:05,352 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:05,352 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:05,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3623f5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:05,356 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:05,356 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,358 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:05,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:05,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:05,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:05,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:05,362 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:05,362 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:05,362 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:05,362 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:05,362 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:05,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,364 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,376 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:05,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:05,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,376 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,376 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:05,376 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,376 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:05,376 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,376 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,476 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:05,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:05,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:05,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:05,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:05,479 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:05,479 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,479 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:05,480 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:05,480 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,480 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,481 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:05,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:05,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:05,483 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:05,483 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,483 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:05,483 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:05,483 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:05,483 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,483 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:05,483 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:05,483 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,483 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:05,484 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:05,484 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,484 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:05,484 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:05,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,485 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:05,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@252ea163[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:05,487 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:05,487 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,489 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,489 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:05,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:05,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:05,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:05,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,491 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:05,491 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,491 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:05,493 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:05,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:05,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:05,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:05,493 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:05,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,496 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,497 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:05,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:05,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,499 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:05,500 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:05,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:05,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507765533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,600 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:05,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:05,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:05,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:05,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:05,604 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:05,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:05,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:05,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:05,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:05,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:05,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:05,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:05,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:05,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:05,609 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:05,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31295517[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:05,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,613 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:05,613 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:05,613 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:05,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:05,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:05,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:05,617 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:05,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:05,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:05,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:05,617 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:05,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:05,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:05,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,625 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:05,625 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:05,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/d1a6c76e7d8549109cf8e5de2b94029a 2023-03-22 17:55:05,725 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:05,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:05,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:05,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:05,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:05,728 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:05,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:05,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:05,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,735 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:05,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:05,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:05,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:05,738 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,738 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:05,738 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:05,738 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:05,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,740 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,740 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,740 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:05,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5434298[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:05,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:05,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,744 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:05,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:05,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:05,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:05,746 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:05,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:05,748 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:05,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:05,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:05,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:05,748 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:05,749 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,749 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,749 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,750 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:05,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:05,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:05,757 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:05,757 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:05,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,857 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:05,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:05,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:05,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:05,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:05,860 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:05,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:05,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:05,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,865 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,865 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:05,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:05,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:05,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:05,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:05,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:05,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:05,868 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7817f04e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:05,869 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:05,869 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:05,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:05,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:05,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:05,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:05,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,875 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:05,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:05,877 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:05,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:05,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:05,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:05,878 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:05,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:05,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:05,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,882 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,882 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:05,882 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:05,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:05,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:05,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,883 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:05,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:05,883 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:05,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:05,983 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:05,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:05,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:05,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:05,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:05,992 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:05,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:05,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:05,992 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:05,993 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,993 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,994 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:05,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,995 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:05,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:05,995 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:05,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:06,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:06,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:06,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:06,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:06,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:06,008 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:06,008 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:06,008 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:06,008 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:06,008 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:06,009 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:06,009 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:06,009 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:06,009 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,009 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:06,009 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:06,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:06,012 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:06,013 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:06,013 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:06,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:06,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28627396[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:06,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:06,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:06,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:06,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:06,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:06,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:06,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:06,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:06,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:06,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:06,018 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:06,018 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:06,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:06,022 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:06,022 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:06,022 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:06,022 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:06,022 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:06,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:06,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,025 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,027 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:06,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:06,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:06,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:06,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:06,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:06,032 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:06,032 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:06,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:06,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507766039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/aa441ca951da4841b6af1044596a95f2 2023-03-22 17:55:06,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/e023a44e7c1b4ae7ad93074933f92f9e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e023a44e7c1b4ae7ad93074933f92f9e 2023-03-22 17:55:06,118 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e023a44e7c1b4ae7ad93074933f92f9e, entries=150, sequenceid=432, filesize=11.9 K 2023-03-22 17:55:06,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/d1a6c76e7d8549109cf8e5de2b94029a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/d1a6c76e7d8549109cf8e5de2b94029a 2023-03-22 17:55:06,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/d1a6c76e7d8549109cf8e5de2b94029a, entries=150, sequenceid=432, filesize=11.9 K 2023-03-22 17:55:06,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/aa441ca951da4841b6af1044596a95f2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aa441ca951da4841b6af1044596a95f2 2023-03-22 17:55:06,131 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aa441ca951da4841b6af1044596a95f2, entries=150, sequenceid=432, filesize=11.9 K 2023-03-22 17:55:06,132 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:06,132 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a0a59fb2216419102dce85f7adab989d in 1257ms, sequenceid=432, compaction requested=true 2023-03-22 17:55:06,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:06,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:06,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:06,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:06,132 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:06,132 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:06,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:06,133 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:06,133 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:06,134 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:55:06,134 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:55:06,134 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:06,134 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:06,134 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/3c55b3ca8d894d23a499e50ab5106ef1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/1c0100ad68d04252b21704617c87cddb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e023a44e7c1b4ae7ad93074933f92f9e] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.5 K 2023-03-22 17:55:06,134 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/a976ca1fbd7d4a4e85018991ac30d44c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c0431b2046fd494d9a3628ca63055a50, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/d1a6c76e7d8549109cf8e5de2b94029a] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.5 K 2023-03-22 17:55:06,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:06,135 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 3c55b3ca8d894d23a499e50ab5106ef1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507700963 2023-03-22 17:55:06,135 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a976ca1fbd7d4a4e85018991ac30d44c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507700963 2023-03-22 17:55:06,135 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1c0100ad68d04252b21704617c87cddb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679507703161 2023-03-22 17:55:06,135 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting c0431b2046fd494d9a3628ca63055a50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679507703161 2023-03-22 17:55:06,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:06,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:06,136 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:06,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:06,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:06,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:06,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:06,137 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e023a44e7c1b4ae7ad93074933f92f9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679507704250 2023-03-22 17:55:06,137 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d1a6c76e7d8549109cf8e5de2b94029a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679507704250 2023-03-22 17:55:06,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:06,138 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:06,138 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:06,138 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:06,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:06,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:06,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:06,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:06,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:06,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:06,146 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:06,146 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:06,146 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:06,146 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:06,146 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-22 17:55:06,146 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:06,147 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:06,147 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:06,147 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:06,147 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:06,147 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:06,147 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:06,147 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:06,176 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/446d6263c2ad4940afa385ccbc2eb8f6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/446d6263c2ad4940afa385ccbc2eb8f6 2023-03-22 17:55:06,184 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/6cc11b2928bd45abb1e32d540b94f680 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/6cc11b2928bd45abb1e32d540b94f680 2023-03-22 17:55:06,185 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into 446d6263c2ad4940afa385ccbc2eb8f6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:06,185 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:06,185 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507706132; duration=0sec 2023-03-22 17:55:06,186 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:06,186 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:06,187 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:06,187 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:55:06,187 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:06,187 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/a28911004b004395b5ad3eedfea272a9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d8e2b0689c484bf0b7e9627f6cffd508, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aa441ca951da4841b6af1044596a95f2] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=36.5 K 2023-03-22 17:55:06,187 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a28911004b004395b5ad3eedfea272a9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1679507700963 2023-03-22 17:55:06,188 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d8e2b0689c484bf0b7e9627f6cffd508, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679507703161 2023-03-22 17:55:06,189 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting aa441ca951da4841b6af1044596a95f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679507704250 2023-03-22 17:55:06,191 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into 6cc11b2928bd45abb1e32d540b94f680(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:06,191 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:06,191 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507706132; duration=0sec 2023-03-22 17:55:06,191 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:06,195 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#285 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:06,594 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/c3532be80df747948c9df5deee1ec485 2023-03-22 17:55:06,612 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/25a35834748c444ea2687df5888b0b33 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/25a35834748c444ea2687df5888b0b33 2023-03-22 17:55:06,617 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into 25a35834748c444ea2687df5888b0b33(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:06,617 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:06,617 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507706132; duration=0sec 2023-03-22 17:55:06,618 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:06,622 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/c401a7b65ea341559feada122fdad388 2023-03-22 17:55:07,037 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/d2df3ddd3fb34d30a37e0541ac84898a 2023-03-22 17:55:07,042 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/c3532be80df747948c9df5deee1ec485 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c3532be80df747948c9df5deee1ec485 2023-03-22 17:55:07,046 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c3532be80df747948c9df5deee1ec485, entries=150, sequenceid=452, filesize=11.9 K 2023-03-22 17:55:07,047 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/c401a7b65ea341559feada122fdad388 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c401a7b65ea341559feada122fdad388 2023-03-22 17:55:07,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:07,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:07,053 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c401a7b65ea341559feada122fdad388, entries=150, sequenceid=452, filesize=11.9 K 2023-03-22 17:55:07,054 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/d2df3ddd3fb34d30a37e0541ac84898a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d2df3ddd3fb34d30a37e0541ac84898a 2023-03-22 17:55:07,060 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d2df3ddd3fb34d30a37e0541ac84898a, entries=150, sequenceid=452, filesize=11.9 K 2023-03-22 17:55:07,061 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=26.84 KB/27480 for a0a59fb2216419102dce85f7adab989d in 915ms, sequenceid=452, compaction requested=false 2023-03-22 17:55:07,061 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:07,061 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:07,062 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:07,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53180a60[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:07,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:07,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,066 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:07,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:07,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:07,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:07,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:07,068 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:07,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:07,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:07,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:07,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:07,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:07,072 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:07,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:07,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:07,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:07,072 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:07,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/bf7f8417c76e43968e3c2b11be45bc47 2023-03-22 17:55:07,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,082 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,082 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:07,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:07,082 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,082 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,083 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,083 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:07,083 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:07,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507707097, max timestamp=1679507707128Num uniques 859; 2023-03-22 17:55:07,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507707097, max timestamp=1679507707128Num uniques 859; 2023-03-22 17:55:07,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:07,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:07,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507707097, max timestamp=1679507707128Num uniques 859; 2023-03-22 17:55:07,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:07,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:07,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507767170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,183 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:07,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:07,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:07,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:07,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:07,186 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:07,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:07,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:07,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,187 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:07,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:07,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:07,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:07,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:07,192 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,192 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,193 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,193 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:07,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd0d519[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:07,194 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:07,194 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,195 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,195 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:07,195 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:07,196 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:07,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:07,197 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,197 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:07,197 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,197 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:07,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:07,200 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:07,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:07,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:07,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:07,200 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:07,202 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,202 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,203 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,203 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,205 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,208 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,208 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,213 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:07,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:07,213 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:07,213 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:07,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:07,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:07,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507767275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:07,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54144 deadline: 1679507767278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,283 DEBUG [Thread-1633] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18275 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:07,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:07,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54132 deadline: 1679507767302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,304 DEBUG [Thread-1631] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:07,313 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:07,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:07,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:07,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:07,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:07,316 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:07,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:07,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:07,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,319 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,319 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:07,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:07,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:07,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:07,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:07,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:07,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:07,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:07,328 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,328 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:07,328 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:07,328 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:07,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:07,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:07,328 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,330 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:07,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b02d94d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:07,332 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:07,332 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,333 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:07,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:07,335 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:07,335 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,335 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,335 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:07,337 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:07,338 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:07,338 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:07,338 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:07,338 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:07,339 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,339 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,339 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:07,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:07,344 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:07,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,344 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:07,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,345 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,345 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:07,445 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:07,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:07,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:07,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:07,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:07,448 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:07,448 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,448 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:07,448 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:07,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,451 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:07,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:07,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:07,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:07,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:07,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:07,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,452 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:07,453 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:07,453 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:07,456 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,456 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:07,456 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:07,456 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:07,461 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,461 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,461 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,461 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:07,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3047f578[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:07,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:07,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,464 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:07,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:07,464 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:07,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:07,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:07,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,465 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:07,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:07,468 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:07,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:07,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:07,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:07,468 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:07,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:07,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:07,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,476 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:07,476 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:07,476 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:07,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507767480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/095771e47eb04562bdf2f1327f22ac80 2023-03-22 17:55:07,576 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:07,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:07,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:07,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:07,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:07,579 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:07,579 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,580 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:07,580 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:07,580 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,580 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,581 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,581 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:07,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:07,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:07,583 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,583 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:07,583 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:07,583 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:07,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:07,585 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,585 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:07,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:07,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@239fb21a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:07,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:07,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,591 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:07,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:07,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:07,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:07,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:07,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,593 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:07,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:07,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:07,596 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:07,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:07,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:07,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:07,596 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:07,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,598 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,598 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:07,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:07,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,602 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:07,602 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:07,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,703 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:07,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:07,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:07,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:07,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:07,706 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:07,706 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,706 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:07,706 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:07,706 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,706 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,709 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,709 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,709 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,709 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:07,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:07,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:07,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:07,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,711 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. started... 2023-03-22 17:55:07,711 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. as already flushing 2023-03-22 17:55:07,711 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:07,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:07,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:07,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:07,712 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,713 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:07,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69adc36d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:07,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:07,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,715 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:07,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:07,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:07,716 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:07,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,717 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:07,717 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,717 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:07,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:07,720 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:07,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:07,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:07,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:07,720 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:07,722 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,722 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,722 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:07,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:07,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:07,734 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:07,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:07,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:07,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:07,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:07,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:07,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:07,742 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:07,742 DEBUG [Thread-1639] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:07,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:07,742 DEBUG [Thread-1639] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:07,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:07,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:07,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54212 deadline: 1679507767785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:07,794 DEBUG [Thread-1642] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:55285 2023-03-22 17:55:07,795 DEBUG [Thread-1642] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:07,796 DEBUG [Thread-1640] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:55285 2023-03-22 17:55:07,796 DEBUG [Thread-1640] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:07,797 DEBUG [Thread-1648] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:55285 2023-03-22 17:55:07,797 DEBUG [Thread-1648] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:07,797 DEBUG [Thread-1646] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:55285 2023-03-22 17:55:07,797 DEBUG [Thread-1646] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:07,799 DEBUG [Thread-1644] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:55285 2023-03-22 17:55:07,799 DEBUG [Thread-1644] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:07,842 DEBUG [Thread-1639] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:07,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:07,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/84fc9cdba4e046949fa64e53e06bc3ff 2023-03-22 17:55:07,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/bf7f8417c76e43968e3c2b11be45bc47 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bf7f8417c76e43968e3c2b11be45bc47 2023-03-22 17:55:07,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bf7f8417c76e43968e3c2b11be45bc47, entries=100, sequenceid=465, filesize=9.5 K 2023-03-22 17:55:07,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/095771e47eb04562bdf2f1327f22ac80 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/095771e47eb04562bdf2f1327f22ac80 2023-03-22 17:55:07,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/095771e47eb04562bdf2f1327f22ac80, entries=100, sequenceid=465, filesize=9.5 K 2023-03-22 17:55:07,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/84fc9cdba4e046949fa64e53e06bc3ff as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/84fc9cdba4e046949fa64e53e06bc3ff 2023-03-22 17:55:07,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/84fc9cdba4e046949fa64e53e06bc3ff, entries=100, sequenceid=465, filesize=9.5 K 2023-03-22 17:55:07,914 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a0a59fb2216419102dce85f7adab989d in 846ms, sequenceid=465, compaction requested=true 2023-03-22 17:55:07,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:07,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:07,915 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:07,915 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:07,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:07,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:07,916 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:07,916 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/A is initiating minor compaction (all files) 2023-03-22 17:55:07,916 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/A in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,916 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:07,916 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/6cc11b2928bd45abb1e32d540b94f680, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c3532be80df747948c9df5deee1ec485, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bf7f8417c76e43968e3c2b11be45bc47] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.2 K 2023-03-22 17:55:07,916 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/B is initiating minor compaction (all files) 2023-03-22 17:55:07,916 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/B in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:07,916 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/446d6263c2ad4940afa385ccbc2eb8f6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c401a7b65ea341559feada122fdad388, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/095771e47eb04562bdf2f1327f22ac80] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.2 K 2023-03-22 17:55:07,916 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 6cc11b2928bd45abb1e32d540b94f680, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679507704250 2023-03-22 17:55:07,917 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 446d6263c2ad4940afa385ccbc2eb8f6, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679507704250 2023-03-22 17:55:07,917 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c3532be80df747948c9df5deee1ec485, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679507704908 2023-03-22 17:55:07,917 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting c401a7b65ea341559feada122fdad388, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679507704908 2023-03-22 17:55:07,917 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting bf7f8417c76e43968e3c2b11be45bc47, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679507707065 2023-03-22 17:55:07,917 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 095771e47eb04562bdf2f1327f22ac80, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679507707065 2023-03-22 17:55:07,928 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#B#compaction#291 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:07,929 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#A#compaction#292 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:08,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:08,291 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-22 17:55:08,291 DEBUG [Thread-1629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:55285 2023-03-22 17:55:08,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:08,291 DEBUG [Thread-1629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:08,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:55:08,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:08,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:55:08,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:08,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:55:08,336 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1fa23117e1af4c5fa9aa414c7d3ac90d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1fa23117e1af4c5fa9aa414c7d3ac90d 2023-03-22 17:55:08,337 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/fb7aafac50db48a49410981d91bf47b4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/fb7aafac50db48a49410981d91bf47b4 2023-03-22 17:55:08,339 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/B of a0a59fb2216419102dce85f7adab989d into 1fa23117e1af4c5fa9aa414c7d3ac90d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:08,339 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:08,339 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/B, priority=13, startTime=1679507707915; duration=0sec 2023-03-22 17:55:08,339 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:08,340 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:08,340 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/A of a0a59fb2216419102dce85f7adab989d into fb7aafac50db48a49410981d91bf47b4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:08,340 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:08,340 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/A, priority=13, startTime=1679507707914; duration=0sec 2023-03-22 17:55:08,340 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:08,340 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:08,341 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): a0a59fb2216419102dce85f7adab989d/C is initiating minor compaction (all files) 2023-03-22 17:55:08,341 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a0a59fb2216419102dce85f7adab989d/C in TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:08,341 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/25a35834748c444ea2687df5888b0b33, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d2df3ddd3fb34d30a37e0541ac84898a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/84fc9cdba4e046949fa64e53e06bc3ff] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp, totalSize=34.2 K 2023-03-22 17:55:08,341 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 25a35834748c444ea2687df5888b0b33, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1679507704250 2023-03-22 17:55:08,341 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d2df3ddd3fb34d30a37e0541ac84898a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1679507704908 2023-03-22 17:55:08,341 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 84fc9cdba4e046949fa64e53e06bc3ff, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679507707065 2023-03-22 17:55:08,346 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): a0a59fb2216419102dce85f7adab989d#C#compaction#294 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:08,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/d2dc9e26310b45e0b3e6a8aff41e1b3b 2023-03-22 17:55:08,755 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/653b9452d343443e9e1c5b03e924b287 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/653b9452d343443e9e1c5b03e924b287 2023-03-22 17:55:08,758 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a0a59fb2216419102dce85f7adab989d/C of a0a59fb2216419102dce85f7adab989d into 653b9452d343443e9e1c5b03e924b287(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:08,758 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:08,758 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d., storeName=a0a59fb2216419102dce85f7adab989d/C, priority=13, startTime=1679507707915; duration=0sec 2023-03-22 17:55:08,759 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:09,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1a1842fc30ea4ec89a90b6a981a00b97 2023-03-22 17:55:09,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/6e82a6fb9ae04e8f833d448463cb7e57 2023-03-22 17:55:09,519 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/d2dc9e26310b45e0b3e6a8aff41e1b3b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d2dc9e26310b45e0b3e6a8aff41e1b3b 2023-03-22 17:55:09,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d2dc9e26310b45e0b3e6a8aff41e1b3b, entries=150, sequenceid=503, filesize=11.9 K 2023-03-22 17:55:09,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/1a1842fc30ea4ec89a90b6a981a00b97 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1a1842fc30ea4ec89a90b6a981a00b97 2023-03-22 17:55:09,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1a1842fc30ea4ec89a90b6a981a00b97, entries=150, sequenceid=503, filesize=11.9 K 2023-03-22 17:55:09,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/6e82a6fb9ae04e8f833d448463cb7e57 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/6e82a6fb9ae04e8f833d448463cb7e57 2023-03-22 17:55:09,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/6e82a6fb9ae04e8f833d448463cb7e57, entries=150, sequenceid=503, filesize=11.9 K 2023-03-22 17:55:09,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=0 B/0 for a0a59fb2216419102dce85f7adab989d in 1240ms, sequenceid=503, compaction requested=false 2023-03-22 17:55:09,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:11,336 DEBUG [Thread-1631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:55285 2023-03-22 17:55:11,337 DEBUG [Thread-1631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:13,968 DEBUG [Thread-1635] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:55285 2023-03-22 17:55:13,968 DEBUG [Thread-1635] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:13,994 DEBUG [Thread-1637] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:55285 2023-03-22 17:55:13,994 DEBUG [Thread-1637] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:17,308 DEBUG [Thread-1633] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:55285 2023-03-22 17:55:17,308 DEBUG [Thread-1633] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:17,308 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 151 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 5648 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 5683 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 5609 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 5636 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 5699 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-22 17:55:17,309 INFO [Listener at localhost/44491] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-22 17:55:17,309 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x112921f2 to 127.0.0.1:55285 2023-03-22 17:55:17,309 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:17,310 INFO [Listener at localhost/44491] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-22 17:55:17,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-22 17:55:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:17,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-22 17:55:17,313 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507717313"}]},"ts":"1679507717313"} 2023-03-22 17:55:17,314 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-22 17:55:17,315 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-22 17:55:17,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a0a59fb2216419102dce85f7adab989d, UNASSIGN}] 2023-03-22 17:55:17,317 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a0a59fb2216419102dce85f7adab989d, UNASSIGN 2023-03-22 17:55:17,318 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=a0a59fb2216419102dce85f7adab989d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:17,318 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507717318"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507717318"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507717318"}]},"ts":"1679507717318"} 2023-03-22 17:55:17,319 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:55:17,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-22 17:55:17,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:17,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a0a59fb2216419102dce85f7adab989d, disabling compactions & flushes 2023-03-22 17:55:17,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:17,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:17,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. after waiting 0 ms 2023-03-22 17:55:17,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:17,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a0a59fb2216419102dce85f7adab989d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-22 17:55:17,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=A 2023-03-22 17:55:17,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:17,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=B 2023-03-22 17:55:17,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:17,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a0a59fb2216419102dce85f7adab989d, store=C 2023-03-22 17:55:17,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:17,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-22 17:55:17,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/79af53812e7c40788223fe1ea493d472 2023-03-22 17:55:17,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-22 17:55:18,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/54db3e4ce67343ef9546d09733cad11e 2023-03-22 17:55:18,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-22 17:55:18,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/f2ebc69451d6492d9403ef5217438e62 2023-03-22 17:55:18,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/A/79af53812e7c40788223fe1ea493d472 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/79af53812e7c40788223fe1ea493d472 2023-03-22 17:55:18,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/79af53812e7c40788223fe1ea493d472, entries=150, sequenceid=513, filesize=11.9 K 2023-03-22 17:55:18,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/B/54db3e4ce67343ef9546d09733cad11e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/54db3e4ce67343ef9546d09733cad11e 2023-03-22 17:55:18,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/54db3e4ce67343ef9546d09733cad11e, entries=150, sequenceid=513, filesize=11.9 K 2023-03-22 17:55:18,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/.tmp/C/f2ebc69451d6492d9403ef5217438e62 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f2ebc69451d6492d9403ef5217438e62 2023-03-22 17:55:18,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f2ebc69451d6492d9403ef5217438e62, entries=150, sequenceid=513, filesize=11.9 K 2023-03-22 17:55:18,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for a0a59fb2216419102dce85f7adab989d in 1241ms, sequenceid=513, compaction requested=true 2023-03-22 17:55:18,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d635cc5156004b36aab3c1b33f8e5efe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5ae697f6ff86417a830b7cd8aca5c66c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/7d9b4c783d5c4037aa088ed634c7d049, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/f700ef4f72bb40cd85188401911c2553, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/14f304a4bb9a46f1b80929d3dbb35a44, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd67ea28784b4a4381f0a8b79497d97d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e8a03b15d12d4fad8199b5fea03b72c4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd172b711bdc4de98c5065b2b537cbf0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/0fcb8681c19f4b4481e19efdf178a2e3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/da1e67c88fff458389641fdf150696a2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c53c8ab8f30d4dbca565a4933062e994, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/444b124fcb2c4bd89519aedeb463333f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e64833f523064b13a6364b204f6ec0f3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/00ab464897594460b92b04d8df6979ec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/35fca7f94e8e407da4720cb2def34afd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2228685e78c64a7db4bf9f2869641ca2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5a3dcae63d524c7a96f81ebf8ef69538, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/ab6861be02374de18383ace2e9c95178, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/a22d15f906e544b19c94af1078b7e975, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/cdea144660e04c66bb9edb2118e666e7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2dbb26d183434e98a8fe5bc7092a5300, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/451ccf1560a54ad29dbef175bfe81f9e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/b896f3c533974e1d8be3e6925e4b15e1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2c3095626b984f59abb1ddfd1d05c471, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/3c55b3ca8d894d23a499e50ab5106ef1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5413be8516534cd0804ad8612356306e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/1c0100ad68d04252b21704617c87cddb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/6cc11b2928bd45abb1e32d540b94f680, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e023a44e7c1b4ae7ad93074933f92f9e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c3532be80df747948c9df5deee1ec485, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bf7f8417c76e43968e3c2b11be45bc47] to archive 2023-03-22 17:55:18,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:18,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d635cc5156004b36aab3c1b33f8e5efe to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d635cc5156004b36aab3c1b33f8e5efe 2023-03-22 17:55:18,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5ae697f6ff86417a830b7cd8aca5c66c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5ae697f6ff86417a830b7cd8aca5c66c 2023-03-22 17:55:18,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/7d9b4c783d5c4037aa088ed634c7d049 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/7d9b4c783d5c4037aa088ed634c7d049 2023-03-22 17:55:18,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/f700ef4f72bb40cd85188401911c2553 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/f700ef4f72bb40cd85188401911c2553 2023-03-22 17:55:18,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/14f304a4bb9a46f1b80929d3dbb35a44 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/14f304a4bb9a46f1b80929d3dbb35a44 2023-03-22 17:55:18,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd67ea28784b4a4381f0a8b79497d97d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd67ea28784b4a4381f0a8b79497d97d 2023-03-22 17:55:18,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e8a03b15d12d4fad8199b5fea03b72c4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e8a03b15d12d4fad8199b5fea03b72c4 2023-03-22 17:55:18,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd172b711bdc4de98c5065b2b537cbf0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bd172b711bdc4de98c5065b2b537cbf0 2023-03-22 17:55:18,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/0fcb8681c19f4b4481e19efdf178a2e3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/0fcb8681c19f4b4481e19efdf178a2e3 2023-03-22 17:55:18,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/da1e67c88fff458389641fdf150696a2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/da1e67c88fff458389641fdf150696a2 2023-03-22 17:55:18,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c53c8ab8f30d4dbca565a4933062e994 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c53c8ab8f30d4dbca565a4933062e994 2023-03-22 17:55:18,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/444b124fcb2c4bd89519aedeb463333f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/444b124fcb2c4bd89519aedeb463333f 2023-03-22 17:55:18,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e64833f523064b13a6364b204f6ec0f3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e64833f523064b13a6364b204f6ec0f3 2023-03-22 17:55:18,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/00ab464897594460b92b04d8df6979ec to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/00ab464897594460b92b04d8df6979ec 2023-03-22 17:55:18,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/35fca7f94e8e407da4720cb2def34afd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/35fca7f94e8e407da4720cb2def34afd 2023-03-22 17:55:18,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2228685e78c64a7db4bf9f2869641ca2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2228685e78c64a7db4bf9f2869641ca2 2023-03-22 17:55:18,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5a3dcae63d524c7a96f81ebf8ef69538 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5a3dcae63d524c7a96f81ebf8ef69538 2023-03-22 17:55:18,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/ab6861be02374de18383ace2e9c95178 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/ab6861be02374de18383ace2e9c95178 2023-03-22 17:55:18,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/a22d15f906e544b19c94af1078b7e975 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/a22d15f906e544b19c94af1078b7e975 2023-03-22 17:55:18,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/cdea144660e04c66bb9edb2118e666e7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/cdea144660e04c66bb9edb2118e666e7 2023-03-22 17:55:18,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2dbb26d183434e98a8fe5bc7092a5300 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2dbb26d183434e98a8fe5bc7092a5300 2023-03-22 17:55:18,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/451ccf1560a54ad29dbef175bfe81f9e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/451ccf1560a54ad29dbef175bfe81f9e 2023-03-22 17:55:18,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/b896f3c533974e1d8be3e6925e4b15e1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/b896f3c533974e1d8be3e6925e4b15e1 2023-03-22 17:55:18,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2c3095626b984f59abb1ddfd1d05c471 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/2c3095626b984f59abb1ddfd1d05c471 2023-03-22 17:55:18,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/3c55b3ca8d894d23a499e50ab5106ef1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/3c55b3ca8d894d23a499e50ab5106ef1 2023-03-22 17:55:18,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5413be8516534cd0804ad8612356306e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/5413be8516534cd0804ad8612356306e 2023-03-22 17:55:18,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/1c0100ad68d04252b21704617c87cddb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/1c0100ad68d04252b21704617c87cddb 2023-03-22 17:55:18,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/6cc11b2928bd45abb1e32d540b94f680 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/6cc11b2928bd45abb1e32d540b94f680 2023-03-22 17:55:18,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e023a44e7c1b4ae7ad93074933f92f9e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/e023a44e7c1b4ae7ad93074933f92f9e 2023-03-22 17:55:18,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c3532be80df747948c9df5deee1ec485 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/c3532be80df747948c9df5deee1ec485 2023-03-22 17:55:18,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bf7f8417c76e43968e3c2b11be45bc47 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/bf7f8417c76e43968e3c2b11be45bc47 2023-03-22 17:55:18,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ce76ba4fc0d04eba995240449fc726d1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/83aafadd9fe74aa5aa6637f07e469ec0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ca4e66c5315a40328b5dc630027895ad, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/949def0e30d349448029433287aca24a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/479db439d8be419f9fa92bb488a98f41, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/15b9eff3edd54f18b69db0285feb4b87, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/8a0f2fd5b84d4d7da4cfd463644bf565, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ed5ce508bffa42968d453f7f65e47059, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f5de1db040e42eb8edcc8ba606c9b3b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/3125366a55434b0fa811905f71efd885, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1e357d1ec2f54d2dad7228e36315c65f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/32abf14ec1a54b94ad674d94b5dab943, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/7cc6441ac53548528f3c2f1c303bd750, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/57d00dbbc539405cb0eabb8c6b79c847, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/f9d8cbeb899d43628159b79d0dcd807d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f0236a618664613a3ce45f3038703ad, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/2d6f14bca789452db9ca49d6724fe1dc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/6507a34877004646abafca99e4000bde, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/dec1ada75ff2490f8cde6f4252114b01, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/774cc840e66c4926b3c3dcfbb341a32a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/edf60a71058b410d82e79970214ac543, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/89b6b61299e24711a9cceff136cfa8fe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1bb3062e455f43d1b783d4855e697e67, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/00bf964e611d446e86ed069abe4b9879, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/a976ca1fbd7d4a4e85018991ac30d44c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/4c803e0d45474852ba0a6e92aad4ea65, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c0431b2046fd494d9a3628ca63055a50, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/446d6263c2ad4940afa385ccbc2eb8f6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/d1a6c76e7d8549109cf8e5de2b94029a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c401a7b65ea341559feada122fdad388, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/095771e47eb04562bdf2f1327f22ac80] to archive 2023-03-22 17:55:18,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:18,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ce76ba4fc0d04eba995240449fc726d1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ce76ba4fc0d04eba995240449fc726d1 2023-03-22 17:55:18,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/83aafadd9fe74aa5aa6637f07e469ec0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/83aafadd9fe74aa5aa6637f07e469ec0 2023-03-22 17:55:18,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ca4e66c5315a40328b5dc630027895ad to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ca4e66c5315a40328b5dc630027895ad 2023-03-22 17:55:18,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/949def0e30d349448029433287aca24a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/949def0e30d349448029433287aca24a 2023-03-22 17:55:18,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/479db439d8be419f9fa92bb488a98f41 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/479db439d8be419f9fa92bb488a98f41 2023-03-22 17:55:18,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/15b9eff3edd54f18b69db0285feb4b87 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/15b9eff3edd54f18b69db0285feb4b87 2023-03-22 17:55:18,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/8a0f2fd5b84d4d7da4cfd463644bf565 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/8a0f2fd5b84d4d7da4cfd463644bf565 2023-03-22 17:55:18,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ed5ce508bffa42968d453f7f65e47059 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/ed5ce508bffa42968d453f7f65e47059 2023-03-22 17:55:18,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f5de1db040e42eb8edcc8ba606c9b3b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f5de1db040e42eb8edcc8ba606c9b3b 2023-03-22 17:55:18,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/3125366a55434b0fa811905f71efd885 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/3125366a55434b0fa811905f71efd885 2023-03-22 17:55:18,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1e357d1ec2f54d2dad7228e36315c65f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1e357d1ec2f54d2dad7228e36315c65f 2023-03-22 17:55:18,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/32abf14ec1a54b94ad674d94b5dab943 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/32abf14ec1a54b94ad674d94b5dab943 2023-03-22 17:55:18,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/7cc6441ac53548528f3c2f1c303bd750 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/7cc6441ac53548528f3c2f1c303bd750 2023-03-22 17:55:18,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/57d00dbbc539405cb0eabb8c6b79c847 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/57d00dbbc539405cb0eabb8c6b79c847 2023-03-22 17:55:18,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/f9d8cbeb899d43628159b79d0dcd807d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/f9d8cbeb899d43628159b79d0dcd807d 2023-03-22 17:55:18,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f0236a618664613a3ce45f3038703ad to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1f0236a618664613a3ce45f3038703ad 2023-03-22 17:55:18,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/2d6f14bca789452db9ca49d6724fe1dc to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/2d6f14bca789452db9ca49d6724fe1dc 2023-03-22 17:55:18,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/6507a34877004646abafca99e4000bde to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/6507a34877004646abafca99e4000bde 2023-03-22 17:55:18,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/dec1ada75ff2490f8cde6f4252114b01 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/dec1ada75ff2490f8cde6f4252114b01 2023-03-22 17:55:18,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/774cc840e66c4926b3c3dcfbb341a32a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/774cc840e66c4926b3c3dcfbb341a32a 2023-03-22 17:55:18,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/edf60a71058b410d82e79970214ac543 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/edf60a71058b410d82e79970214ac543 2023-03-22 17:55:18,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/89b6b61299e24711a9cceff136cfa8fe to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/89b6b61299e24711a9cceff136cfa8fe 2023-03-22 17:55:18,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1bb3062e455f43d1b783d4855e697e67 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1bb3062e455f43d1b783d4855e697e67 2023-03-22 17:55:18,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/00bf964e611d446e86ed069abe4b9879 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/00bf964e611d446e86ed069abe4b9879 2023-03-22 17:55:18,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/a976ca1fbd7d4a4e85018991ac30d44c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/a976ca1fbd7d4a4e85018991ac30d44c 2023-03-22 17:55:18,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/4c803e0d45474852ba0a6e92aad4ea65 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/4c803e0d45474852ba0a6e92aad4ea65 2023-03-22 17:55:18,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c0431b2046fd494d9a3628ca63055a50 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c0431b2046fd494d9a3628ca63055a50 2023-03-22 17:55:18,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/446d6263c2ad4940afa385ccbc2eb8f6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/446d6263c2ad4940afa385ccbc2eb8f6 2023-03-22 17:55:18,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/d1a6c76e7d8549109cf8e5de2b94029a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/d1a6c76e7d8549109cf8e5de2b94029a 2023-03-22 17:55:18,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c401a7b65ea341559feada122fdad388 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/c401a7b65ea341559feada122fdad388 2023-03-22 17:55:18,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/095771e47eb04562bdf2f1327f22ac80 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/095771e47eb04562bdf2f1327f22ac80 2023-03-22 17:55:18,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3596b34bc076471fbf5ed2b372b507c4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b10bb5feba0e49418a0b3fb61cb00626, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/01147d9d2c024ca7a1594ae0adecf93f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/651506b69bf846f8b90bed189f1ef42c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/244c91248c1f475bbd4f1696538a6746, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/310429eddd7b419084d9e6dd36c6eef5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/32ac4d33ce844de4b45057c36c4a4a21, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3ea7624175454c4ca6684976ba13736e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1b3966604f54345a741cc4d5eae4960, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/728e5fb148bf49bb8d9a9b9111aa9e63, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/8e44e9851f1942a2b7be37aed3f22a03, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/38543affcb574400b5e8b097e41abab4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/60e12e43cf1b40798631cfbc43fa8b9a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430fa73eca644357a6a36674ba1ff19a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1def152e4b24e789cba417c08641d10, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/511169be58634b91940333a1b8ab2bd0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/cf8e8bf2cc084a1d981e1824a81aa95f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430931344907493cb06767c3d39fa69e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d1a8fe654044428b870c5e566bed7bce, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e428b7957912434096ff5c0d2364ee43, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/50d2e3cf08354ccb89ec74cb14415a96, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f1b2d17d931f4a588ad67bd3b6de0099, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/5bafa8daac6043e3a162adcaf495eae2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aba3726d3e794e9eac68b5532fa05e94, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/a28911004b004395b5ad3eedfea272a9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e7a92f9d904542f1ae55e92b73105e88, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d8e2b0689c484bf0b7e9627f6cffd508, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/25a35834748c444ea2687df5888b0b33, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aa441ca951da4841b6af1044596a95f2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d2df3ddd3fb34d30a37e0541ac84898a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/84fc9cdba4e046949fa64e53e06bc3ff] to archive 2023-03-22 17:55:18,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:18,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3596b34bc076471fbf5ed2b372b507c4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3596b34bc076471fbf5ed2b372b507c4 2023-03-22 17:55:18,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b10bb5feba0e49418a0b3fb61cb00626 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b10bb5feba0e49418a0b3fb61cb00626 2023-03-22 17:55:18,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/01147d9d2c024ca7a1594ae0adecf93f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/01147d9d2c024ca7a1594ae0adecf93f 2023-03-22 17:55:18,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/651506b69bf846f8b90bed189f1ef42c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/651506b69bf846f8b90bed189f1ef42c 2023-03-22 17:55:18,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/244c91248c1f475bbd4f1696538a6746 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/244c91248c1f475bbd4f1696538a6746 2023-03-22 17:55:18,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/310429eddd7b419084d9e6dd36c6eef5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/310429eddd7b419084d9e6dd36c6eef5 2023-03-22 17:55:18,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/32ac4d33ce844de4b45057c36c4a4a21 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/32ac4d33ce844de4b45057c36c4a4a21 2023-03-22 17:55:18,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3ea7624175454c4ca6684976ba13736e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/3ea7624175454c4ca6684976ba13736e 2023-03-22 17:55:18,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1b3966604f54345a741cc4d5eae4960 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1b3966604f54345a741cc4d5eae4960 2023-03-22 17:55:18,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/728e5fb148bf49bb8d9a9b9111aa9e63 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/728e5fb148bf49bb8d9a9b9111aa9e63 2023-03-22 17:55:18,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/8e44e9851f1942a2b7be37aed3f22a03 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/8e44e9851f1942a2b7be37aed3f22a03 2023-03-22 17:55:18,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/38543affcb574400b5e8b097e41abab4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/38543affcb574400b5e8b097e41abab4 2023-03-22 17:55:18,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/60e12e43cf1b40798631cfbc43fa8b9a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/60e12e43cf1b40798631cfbc43fa8b9a 2023-03-22 17:55:18,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430fa73eca644357a6a36674ba1ff19a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430fa73eca644357a6a36674ba1ff19a 2023-03-22 17:55:18,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1def152e4b24e789cba417c08641d10 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/b1def152e4b24e789cba417c08641d10 2023-03-22 17:55:18,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/511169be58634b91940333a1b8ab2bd0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/511169be58634b91940333a1b8ab2bd0 2023-03-22 17:55:18,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/cf8e8bf2cc084a1d981e1824a81aa95f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/cf8e8bf2cc084a1d981e1824a81aa95f 2023-03-22 17:55:18,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430931344907493cb06767c3d39fa69e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/430931344907493cb06767c3d39fa69e 2023-03-22 17:55:18,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d1a8fe654044428b870c5e566bed7bce to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d1a8fe654044428b870c5e566bed7bce 2023-03-22 17:55:18,815 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e428b7957912434096ff5c0d2364ee43 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e428b7957912434096ff5c0d2364ee43 2023-03-22 17:55:18,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/50d2e3cf08354ccb89ec74cb14415a96 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/50d2e3cf08354ccb89ec74cb14415a96 2023-03-22 17:55:18,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f1b2d17d931f4a588ad67bd3b6de0099 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f1b2d17d931f4a588ad67bd3b6de0099 2023-03-22 17:55:18,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/5bafa8daac6043e3a162adcaf495eae2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/5bafa8daac6043e3a162adcaf495eae2 2023-03-22 17:55:18,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aba3726d3e794e9eac68b5532fa05e94 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aba3726d3e794e9eac68b5532fa05e94 2023-03-22 17:55:18,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/a28911004b004395b5ad3eedfea272a9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/a28911004b004395b5ad3eedfea272a9 2023-03-22 17:55:18,821 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e7a92f9d904542f1ae55e92b73105e88 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/e7a92f9d904542f1ae55e92b73105e88 2023-03-22 17:55:18,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d8e2b0689c484bf0b7e9627f6cffd508 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d8e2b0689c484bf0b7e9627f6cffd508 2023-03-22 17:55:18,823 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/25a35834748c444ea2687df5888b0b33 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/25a35834748c444ea2687df5888b0b33 2023-03-22 17:55:18,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aa441ca951da4841b6af1044596a95f2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/aa441ca951da4841b6af1044596a95f2 2023-03-22 17:55:18,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d2df3ddd3fb34d30a37e0541ac84898a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/d2df3ddd3fb34d30a37e0541ac84898a 2023-03-22 17:55:18,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/84fc9cdba4e046949fa64e53e06bc3ff to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/84fc9cdba4e046949fa64e53e06bc3ff 2023-03-22 17:55:18,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/recovered.edits/516.seqid, newMaxSeqId=516, maxSeqId=1 2023-03-22 17:55:18,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d. 2023-03-22 17:55:18,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a0a59fb2216419102dce85f7adab989d: 2023-03-22 17:55:18,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:18,833 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=a0a59fb2216419102dce85f7adab989d, regionState=CLOSED 2023-03-22 17:55:18,833 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507718833"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507718833"}]},"ts":"1679507718833"} 2023-03-22 17:55:18,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-22 17:55:18,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure a0a59fb2216419102dce85f7adab989d, server=jenkins-hbase4.apache.org,34403,1679507617723 in 1.5150 sec 2023-03-22 17:55:18,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-22 17:55:18,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a0a59fb2216419102dce85f7adab989d, UNASSIGN in 1.5190 sec 2023-03-22 17:55:18,837 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507718837"}]},"ts":"1679507718837"} 2023-03-22 17:55:18,838 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-22 17:55:18,841 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-22 17:55:18,842 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5310 sec 2023-03-22 17:55:19,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-22 17:55:19,417 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-22 17:55:19,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-22 17:55:19,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:19,419 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-22 17:55:19,420 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:19,424 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:19,426 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/recovered.edits] 2023-03-22 17:55:19,428 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/79af53812e7c40788223fe1ea493d472 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/79af53812e7c40788223fe1ea493d472 2023-03-22 17:55:19,429 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d2dc9e26310b45e0b3e6a8aff41e1b3b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/d2dc9e26310b45e0b3e6a8aff41e1b3b 2023-03-22 17:55:19,430 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/fb7aafac50db48a49410981d91bf47b4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/A/fb7aafac50db48a49410981d91bf47b4 2023-03-22 17:55:19,432 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1a1842fc30ea4ec89a90b6a981a00b97 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1a1842fc30ea4ec89a90b6a981a00b97 2023-03-22 17:55:19,433 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1fa23117e1af4c5fa9aa414c7d3ac90d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/1fa23117e1af4c5fa9aa414c7d3ac90d 2023-03-22 17:55:19,434 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/54db3e4ce67343ef9546d09733cad11e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/B/54db3e4ce67343ef9546d09733cad11e 2023-03-22 17:55:19,436 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/653b9452d343443e9e1c5b03e924b287 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/653b9452d343443e9e1c5b03e924b287 2023-03-22 17:55:19,437 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/6e82a6fb9ae04e8f833d448463cb7e57 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/6e82a6fb9ae04e8f833d448463cb7e57 2023-03-22 17:55:19,438 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f2ebc69451d6492d9403ef5217438e62 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/C/f2ebc69451d6492d9403ef5217438e62 2023-03-22 17:55:19,440 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/recovered.edits/516.seqid to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d/recovered.edits/516.seqid 2023-03-22 17:55:19,440 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/a0a59fb2216419102dce85f7adab989d 2023-03-22 17:55:19,440 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:55:19,442 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:19,444 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-22 17:55:19,445 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-22 17:55:19,446 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:19,446 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-22 17:55:19,446 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679507719446"}]},"ts":"9223372036854775807"} 2023-03-22 17:55:19,447 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-22 17:55:19,447 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a0a59fb2216419102dce85f7adab989d, NAME => 'TestAcidGuarantees,,1679507685618.a0a59fb2216419102dce85f7adab989d.', STARTKEY => '', ENDKEY => ''}] 2023-03-22 17:55:19,447 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-22 17:55:19,447 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679507719447"}]},"ts":"9223372036854775807"} 2023-03-22 17:55:19,448 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-22 17:55:19,450 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:19,450 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-03-22 17:55:19,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-22 17:55:19,521 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-22 17:55:19,532 INFO [Listener at localhost/44491] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=524 (was 536), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=283 (was 319), ProcessCount=169 (was 169), AvailableMemoryMB=5876 (was 6013) 2023-03-22 17:55:19,542 INFO [Listener at localhost/44491] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=524, MaxFileDescriptor=60000, SystemLoadAverage=283, ProcessCount=169, AvailableMemoryMB=5876 2023-03-22 17:55:19,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:55:19,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:55:19,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:19,546 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-22 17:55:19,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-22 17:55:19,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-22 17:55:19,547 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-22 17:55:19,548 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:19,548 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 empty. 2023-03-22 17:55:19,549 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:19,549 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:55:19,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-22 17:55:19,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-22 17:55:19,955 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:55:19,957 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2343ed3ff2707ca3be5523fce8688282, NAME => 'TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp 2023-03-22 17:55:20,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-22 17:55:20,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:55:20,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2343ed3ff2707ca3be5523fce8688282, disabling compactions & flushes 2023-03-22 17:55:20,363 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:20,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:20,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. after waiting 0 ms 2023-03-22 17:55:20,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:20,363 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:20,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:20,365 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-22 17:55:20,366 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507720365"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507720365"}]},"ts":"1679507720365"} 2023-03-22 17:55:20,366 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-22 17:55:20,367 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-22 17:55:20,367 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507720367"}]},"ts":"1679507720367"} 2023-03-22 17:55:20,368 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-22 17:55:20,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, ASSIGN}] 2023-03-22 17:55:20,372 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, ASSIGN 2023-03-22 17:55:20,372 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=false 2023-03-22 17:55:20,523 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=2343ed3ff2707ca3be5523fce8688282, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:20,524 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507720523"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507720523"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507720523"}]},"ts":"1679507720523"} 2023-03-22 17:55:20,525 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:55:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-22 17:55:20,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:20,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2343ed3ff2707ca3be5523fce8688282, NAME => 'TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:55:20,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:55:20,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,681 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,682 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A 2023-03-22 17:55:20,682 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A 2023-03-22 17:55:20,682 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:20,683 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2343ed3ff2707ca3be5523fce8688282 columnFamilyName A 2023-03-22 17:55:20,683 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(310): Store=2343ed3ff2707ca3be5523fce8688282/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:20,683 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,684 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B 2023-03-22 17:55:20,684 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B 2023-03-22 17:55:20,684 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:20,684 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2343ed3ff2707ca3be5523fce8688282 columnFamilyName B 2023-03-22 17:55:20,685 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(310): Store=2343ed3ff2707ca3be5523fce8688282/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:20,685 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,686 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C 2023-03-22 17:55:20,686 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C 2023-03-22 17:55:20,686 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:20,686 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2343ed3ff2707ca3be5523fce8688282 columnFamilyName C 2023-03-22 17:55:20,686 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(310): Store=2343ed3ff2707ca3be5523fce8688282/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:20,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:20,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:55:20,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:20,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:55:20,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2343ed3ff2707ca3be5523fce8688282; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75273134, jitterRate=0.1216571033000946}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:55:20,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:20,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., pid=37, masterSystemTime=1679507720676 2023-03-22 17:55:20,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:20,695 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:20,695 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=2343ed3ff2707ca3be5523fce8688282, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:20,695 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507720695"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507720695"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507720695"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507720695"}]},"ts":"1679507720695"} 2023-03-22 17:55:20,697 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-22 17:55:20,697 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 in 171 msec 2023-03-22 17:55:20,698 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-22 17:55:20,698 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, ASSIGN in 327 msec 2023-03-22 17:55:20,698 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-22 17:55:20,698 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507720698"}]},"ts":"1679507720698"} 2023-03-22 17:55:20,699 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-22 17:55:20,701 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-22 17:55:20,702 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-03-22 17:55:21,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-22 17:55:21,651 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-22 17:55:21,652 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x1b77d6d7 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:21,656 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4473f45c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:21,658 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:21,660 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40222, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:21,661 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-22 17:55:21,662 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48992, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-22 17:55:21,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:55:21,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:55:21,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:22,075 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-22 17:55:22,075 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:55:22,076 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-22 17:55:22,077 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-22 17:55:22,080 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, REOPEN/MOVE}] 2023-03-22 17:55:22,080 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, REOPEN/MOVE 2023-03-22 17:55:22,081 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2343ed3ff2707ca3be5523fce8688282, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,081 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507722081"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507722081"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507722081"}]},"ts":"1679507722081"} 2023-03-22 17:55:22,082 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:55:22,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2343ed3ff2707ca3be5523fce8688282, disabling compactions & flushes 2023-03-22 17:55:22,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. after waiting 0 ms 2023-03-22 17:55:22,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-22 17:55:22,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:22,238 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 2343ed3ff2707ca3be5523fce8688282 to self. 2023-03-22 17:55:22,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,239 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2343ed3ff2707ca3be5523fce8688282, regionState=CLOSED 2023-03-22 17:55:22,239 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507722239"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507722239"}]},"ts":"1679507722239"} 2023-03-22 17:55:22,241 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-22 17:55:22,241 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 in 159 msec 2023-03-22 17:55:22,241 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=true 2023-03-22 17:55:22,392 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2343ed3ff2707ca3be5523fce8688282, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,392 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507722392"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507722392"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507722392"}]},"ts":"1679507722392"} 2023-03-22 17:55:22,393 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:55:22,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2343ed3ff2707ca3be5523fce8688282, NAME => 'TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:55:22,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:55:22,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,549 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,550 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A 2023-03-22 17:55:22,550 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A 2023-03-22 17:55:22,551 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:22,551 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2343ed3ff2707ca3be5523fce8688282 columnFamilyName A 2023-03-22 17:55:22,552 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(310): Store=2343ed3ff2707ca3be5523fce8688282/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:22,552 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,553 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B 2023-03-22 17:55:22,553 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B 2023-03-22 17:55:22,553 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:22,553 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2343ed3ff2707ca3be5523fce8688282 columnFamilyName B 2023-03-22 17:55:22,554 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(310): Store=2343ed3ff2707ca3be5523fce8688282/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:22,554 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,554 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C 2023-03-22 17:55:22,554 DEBUG [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C 2023-03-22 17:55:22,555 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:22,555 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2343ed3ff2707ca3be5523fce8688282 columnFamilyName C 2023-03-22 17:55:22,555 INFO [StoreOpener-2343ed3ff2707ca3be5523fce8688282-1] regionserver.HStore(310): Store=2343ed3ff2707ca3be5523fce8688282/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:22,555 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,556 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:55:22,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,560 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2343ed3ff2707ca3be5523fce8688282; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71639160, jitterRate=0.06750667095184326}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:55:22,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:22,560 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., pid=42, masterSystemTime=1679507722545 2023-03-22 17:55:22,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,562 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2343ed3ff2707ca3be5523fce8688282, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,562 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507722562"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507722562"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507722562"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507722562"}]},"ts":"1679507722562"} 2023-03-22 17:55:22,564 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-22 17:55:22,564 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 in 170 msec 2023-03-22 17:55:22,565 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-22 17:55:22,565 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, REOPEN/MOVE in 484 msec 2023-03-22 17:55:22,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-22 17:55:22,567 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-03-22 17:55:22,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 903 msec 2023-03-22 17:55:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-22 17:55:22,569 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x6434ce29 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,575 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41507d16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,575 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,579 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,579 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,582 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,583 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,586 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,586 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,590 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,590 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,594 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,594 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,597 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,597 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,602 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,603 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,607 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,608 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:22,611 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:22,618 DEBUG [hconnection-0x1c88534-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,618 DEBUG [hconnection-0x36399ef7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,619 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40226, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,619 DEBUG [hconnection-0x59a7a632-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,621 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40234, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,621 DEBUG [hconnection-0x71dc531b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,622 DEBUG [hconnection-0x2913aa50-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:22,622 DEBUG [hconnection-0x62dbf2c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,622 DEBUG [hconnection-0x64f72d72-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,622 DEBUG [hconnection-0x292a2e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,623 DEBUG [hconnection-0x1452b546-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,623 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40258, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,624 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,628 DEBUG [hconnection-0x26f3714c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:22,630 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40250, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,630 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40260, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,631 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40308, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,631 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40294, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,631 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40272, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,631 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40302, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:22,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:22,639 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:22,639 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:22,639 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:22,639 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:22,639 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:22,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,645 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:22,645 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:22,645 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,645 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:22,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:22,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:22,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:22,650 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:22,650 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:22,650 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:22,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:22,650 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,650 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:22,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:22,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,661 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:22,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f8a7a4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:22,665 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:22,665 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,666 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:22,666 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,666 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:22,668 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:22,668 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:22,668 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,668 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:22,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:22,670 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:22,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:22,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:22,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:22,670 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:22,672 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,672 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,672 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,672 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,677 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:22,677 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:22,677 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:22,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:22,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:22,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:22,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:22,677 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:22,677 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:22,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:22,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,680 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507782676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507722675, max timestamp=1679507722681Num uniques 859; 2023-03-22 17:55:22,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507722675, max timestamp=1679507722681Num uniques 859; 2023-03-22 17:55:22,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:22,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:22,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507782677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507722675, max timestamp=1679507722681Num uniques 859; 2023-03-22 17:55:22,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:22,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507782707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507782709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507782711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,777 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:22,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:22,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:22,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:22,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:22,782 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:22,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:22,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:22,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:22,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,784 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:22,784 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:22,784 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,784 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:22,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:22,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:22,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:22,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:22,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:22,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:22,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:22,785 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,785 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:22,786 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:22,786 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:22,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,788 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,788 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:22,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507782784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:22,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b1b7076[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:22,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:22,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507782786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,791 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:22,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:22,791 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:22,793 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:22,795 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:22,795 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:22,795 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:22,795 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:22,795 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:22,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:22,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:22,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,800 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:22,800 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:22,800 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,800 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,801 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,801 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,801 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,801 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,800 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:22,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:22,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:22,800 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:22,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:22,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:22,801 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:22,801 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:22,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507782812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507782813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507782815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,901 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:22,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:22,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:22,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:22,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:22,905 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:22,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:22,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:22,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:22,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,907 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:22,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:22,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:22,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:22,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:22,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:22,914 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,914 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:22,914 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:22,914 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:22,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:22,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254a126d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:22,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:22,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:22,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:22,922 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:22,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:22,924 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:22,924 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:22,924 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:22,924 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:22,924 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:22,926 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,926 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:22,926 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:22,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:22,926 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:22,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:22,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:22,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:22,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:22,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:22,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:22,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:22,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,931 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:22,931 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:22,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:22,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:22,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507782990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:22,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507782990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507783019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507783021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507783021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,031 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:23,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:23,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:23,035 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:23,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:23,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:23,036 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,036 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,037 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,037 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,037 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,037 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:23,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:23,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:23,038 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:23,038 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,038 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:23,039 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,039 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:23,039 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:23,039 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:23,041 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:23,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@227b0b02[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:23,043 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:23,043 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,044 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:23,044 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,044 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:23,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:23,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:23,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:23,052 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:23,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:23,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:23,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:23,052 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:23,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,055 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,059 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:23,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,059 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:23,060 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:23,067 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322869447db811c4e2ba0c1f271dbfd8fad to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322869447db811c4e2ba0c1f271dbfd8fad 2023-03-22 17:55:23,160 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:23,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:23,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:23,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:23,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:23,163 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:23,163 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,163 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:23,163 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:23,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,165 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:23,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:23,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:23,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,167 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:23,168 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,168 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:23,168 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:23,168 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:23,168 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:23,170 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,170 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,171 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,171 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:23,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@130f5d6e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:23,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:23,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,175 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:23,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:23,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:23,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:23,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:23,179 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:23,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:23,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:23,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:23,179 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:23,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:23,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,185 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:23,186 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:23,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:23,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,286 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:23,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:23,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:23,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:23,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:23,289 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:23,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:23,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:23,290 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,290 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,291 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,292 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,292 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:23,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:23,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:23,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:23,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:23,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:23,293 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:23,294 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,294 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:23,294 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:23,294 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:23,294 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:23,295 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,295 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@333707b9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:23,298 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:23,298 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,299 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:23,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:23,300 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,300 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:23,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:23,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507783297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,302 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,302 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507783299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:23,305 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:23,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:23,306 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:23,306 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:23,306 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:23,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:23,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:23,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,312 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:23,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,312 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:23,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507783324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507783324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507783325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,412 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:23,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:23,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:23,416 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:23,416 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,416 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:23,416 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:23,417 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,417 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,418 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,418 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,419 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,419 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:23,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:23,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:23,424 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,424 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:23,424 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:23,424 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:23,427 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,427 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,427 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,427 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:23,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@469bb2c8[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:23,429 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:23,429 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:23,430 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,430 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:23,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:23,434 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:23,434 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:23,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:23,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:23,435 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:23,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,440 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,443 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:23,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:23,445 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:23,445 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:23,472 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/0b3b6df8495742ee85a266ad2e906962 2023-03-22 17:55:23,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/043afd621392438da094309a544f24c0 2023-03-22 17:55:23,545 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:23,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:23,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:23,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:23,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:23,548 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:23,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:23,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:23,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:23,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:23,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:23,551 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,551 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:23,551 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:23,551 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,552 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:23,552 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:23,553 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:23,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc29202[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:23,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:23,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,557 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:23,557 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,557 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:23,559 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:23,559 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,559 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,559 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:23,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:23,561 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:23,561 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:23,561 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:23,561 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:23,561 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:23,562 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,562 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,563 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,567 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,567 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:23,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:23,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,567 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:23,567 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,567 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:23,567 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:23,567 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,667 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:23,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:23,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:23,670 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:23,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:23,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:23,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:23,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:23,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:23,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:23,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:23,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:23,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:23,674 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,675 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:23,675 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:23,675 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:23,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@554f4cd9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:23,678 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:23,678 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,680 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:23,680 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:23,680 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:23,680 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:23,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:23,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:23,683 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:23,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:23,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:23,683 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:23,683 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:23,685 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,685 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,687 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,690 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,690 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,690 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,690 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:23,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:23,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,690 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,691 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:23,691 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:23,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,791 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:23,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:23,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:23,794 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:23,794 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,794 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:23,794 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:23,795 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,795 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:23,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:23,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:23,798 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,798 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:23,798 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:23,798 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:23,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:23,800 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:23,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:23,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e4e9306[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:23,801 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,806 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:23,806 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,806 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:23,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:23,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:23,808 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,808 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:23,808 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,808 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507783805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507783806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:23,810 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:23,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:23,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:23,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:23,810 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:23,812 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,812 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:23,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,812 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:23,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,814 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:23,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,819 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,819 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:23,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,819 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:23,819 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:23,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:23,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507783831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507783831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507783832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,893 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/d3d0ee4992d846e9bd926a689697a5b3 2023-03-22 17:55:23,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/0b3b6df8495742ee85a266ad2e906962 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/0b3b6df8495742ee85a266ad2e906962 2023-03-22 17:55:23,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/0b3b6df8495742ee85a266ad2e906962, entries=150, sequenceid=15, filesize=25.1 K 2023-03-22 17:55:23,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/043afd621392438da094309a544f24c0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/043afd621392438da094309a544f24c0 2023-03-22 17:55:23,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/043afd621392438da094309a544f24c0, entries=150, sequenceid=15, filesize=11.6 K 2023-03-22 17:55:23,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/d3d0ee4992d846e9bd926a689697a5b3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d3d0ee4992d846e9bd926a689697a5b3 2023-03-22 17:55:23,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d3d0ee4992d846e9bd926a689697a5b3, entries=150, sequenceid=15, filesize=11.6 K 2023-03-22 17:55:23,912 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2343ed3ff2707ca3be5523fce8688282 in 1272ms, sequenceid=15, compaction requested=false 2023-03-22 17:55:23,912 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:23,920 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:23,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:23,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:23,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:23,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:23,924 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:23,924 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,924 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:23,924 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:23,926 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,926 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:23,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:23,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:23,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:23,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:23,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:23,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:23,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:23,929 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:23,929 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:23,929 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:23,929 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:23,930 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-22 17:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:24,344 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322c785aac699884547a452563cf6b8f7d5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c785aac699884547a452563cf6b8f7d5 2023-03-22 17:55:24,361 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-22 17:55:24,749 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/abf29c70133f43e2a5cd42d5a5e2cd0d 2023-03-22 17:55:24,766 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/91f7f1776bfd4b12ae2b2b8ac088b782 2023-03-22 17:55:24,777 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/6f6d6b193e894d31b229d7a6d615dfc0 2023-03-22 17:55:24,783 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/abf29c70133f43e2a5cd42d5a5e2cd0d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/abf29c70133f43e2a5cd42d5a5e2cd0d 2023-03-22 17:55:24,786 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/abf29c70133f43e2a5cd42d5a5e2cd0d, entries=150, sequenceid=51, filesize=25.1 K 2023-03-22 17:55:24,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/91f7f1776bfd4b12ae2b2b8ac088b782 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/91f7f1776bfd4b12ae2b2b8ac088b782 2023-03-22 17:55:24,791 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/91f7f1776bfd4b12ae2b2b8ac088b782, entries=150, sequenceid=51, filesize=11.6 K 2023-03-22 17:55:24,791 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/6f6d6b193e894d31b229d7a6d615dfc0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/6f6d6b193e894d31b229d7a6d615dfc0 2023-03-22 17:55:24,798 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/6f6d6b193e894d31b229d7a6d615dfc0, entries=150, sequenceid=51, filesize=11.6 K 2023-03-22 17:55:24,798 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 2343ed3ff2707ca3be5523fce8688282 in 868ms, sequenceid=51, compaction requested=false 2023-03-22 17:55:24,798 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:24,798 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:24,798 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,800 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@455877f5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:24,802 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:24,802 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:24,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:24,804 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,804 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:24,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:24,806 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:24,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:24,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:24,809 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:24,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:24,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:24,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:24,809 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:24,811 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,811 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:24,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,811 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:24,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:24,811 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:24,813 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:24,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:24,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:24,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:24,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:24,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:24,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:24,816 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:24,816 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:24,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:24,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:24,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:24,816 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,816 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,816 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:24,895 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:24,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:24,916 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:24,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:24,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:24,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:24,928 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:24,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:24,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:24,928 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:24,929 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,929 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:24,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:24,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:24,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:24,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:24,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:24,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:24,931 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:24,931 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:24,931 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:24,932 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:24,932 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:24,932 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:24,932 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:24,937 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da2bfde[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:24,940 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:24,940 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,941 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:24,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:24,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:24,948 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:24,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:24,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:24,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:24,948 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:24,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:24,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,951 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,953 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:24,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507724935, max timestamp=1679507724954Num uniques 859; 2023-03-22 17:55:24,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:55:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:24,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:24,959 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:24,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:24,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:24,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,959 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:24,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:24,959 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:24,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:24,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:24,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:24,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:24,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:24,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507724936, max timestamp=1679507724957Num uniques 859; 2023-03-22 17:55:24,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:24,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679507724936, max timestamp=1679507724954Num uniques 859; 2023-03-22 17:55:24,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:24,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:24,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507784987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:24,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:24,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507784992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507784995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507784996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507784997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,060 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:25,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:25,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:25,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:25,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:25,063 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:25,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:25,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:25,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:25,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:25,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:25,067 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,067 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:25,067 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:25,067 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:25,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,069 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42241f37[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:25,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:25,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,072 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:25,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:25,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:25,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:25,077 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:25,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:25,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:25,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:25,077 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:25,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,079 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,081 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,087 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:25,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:25,087 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:25,087 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:25,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,087 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:25,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507785098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507785098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507785104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507785105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507785106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,188 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:25,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:25,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:25,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:25,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:25,208 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:25,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:25,208 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:25,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,211 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,211 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:25,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:25,214 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:25,214 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,214 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:25,220 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,220 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:25,220 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:25,220 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:25,222 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,223 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,223 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,223 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:25,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@685872d6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:25,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:25,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:25,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:25,229 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:25,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:25,232 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:25,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:25,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:25,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:25,232 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:25,235 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,235 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,235 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,236 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,239 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:25,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:25,243 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:25,243 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:25,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507785304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507785304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507785311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507785311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507785312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,330 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303226ab3e1f0aab54e36befff4b9c096c787 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303226ab3e1f0aab54e36befff4b9c096c787 2023-03-22 17:55:25,343 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:25,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:25,346 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:25,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:25,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:25,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,346 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,348 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:25,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:25,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:25,350 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,350 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:25,350 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:25,350 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:25,350 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:25,351 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,352 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eb947f7[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:25,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:25,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:25,356 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,356 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:25,358 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:25,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:25,360 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:25,360 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:25,360 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:25,361 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:25,361 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:25,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,363 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,364 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:25,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:25,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,366 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:25,366 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:25,466 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:25,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:25,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:25,470 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:25,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:25,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:25,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,470 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,473 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,474 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:25,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:25,476 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,476 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:25,476 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:25,476 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:25,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:25,478 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169d59b1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:25,479 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:25,479 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,480 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:25,481 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,481 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:25,482 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:25,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:25,484 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:25,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:25,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:25,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:25,485 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:25,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,488 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:25,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:25,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,491 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:25,491 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:25,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:25,591 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:25,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:25,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:25,596 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:25,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:25,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:25,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,596 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:25,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:25,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:25,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:25,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:25,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,609 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74a221f2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:25,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:25,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:25,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:25,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:25,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:25,615 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507785614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:25,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:25,619 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:25,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:25,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:25,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:25,619 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:25,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507785615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507785615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507785616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:25,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507785618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:25,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:25,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:25,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,626 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:25,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,626 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:25,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,726 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:25,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:25,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:25,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:25,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:25,730 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:25,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:25,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:25,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,732 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:25,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:25,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:25,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:25,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:25,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:25,734 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,734 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:25,734 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:25,734 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,734 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:25,734 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:25,734 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,734 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:25,734 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:25,735 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/5da3fd8bbb374a16a5b5b29ebd30c54a 2023-03-22 17:55:25,735 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:25,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fc73372[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:25,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:25,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:25,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:25,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:25,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:25,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:25,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:25,744 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:25,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:25,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:25,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:25,744 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:25,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,746 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:25,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:25,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,751 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:25,751 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:25,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,851 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:25,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:25,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:25,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:25,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:25,855 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:25,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:25,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:25,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,856 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,856 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,857 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,857 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:25,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:25,858 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,858 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:25,858 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:25,858 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:25,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:25,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,861 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,861 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,861 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7be54182[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:25,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:25,863 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:25,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:25,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:25,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:25,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:25,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:25,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:25,867 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:25,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:25,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:25,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:25,868 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:25,869 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,869 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,869 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,869 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,873 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,873 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:25,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:25,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:25,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,874 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:25,874 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:25,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,974 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:25,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:25,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:25,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:25,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:25,978 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:25,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:25,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:25,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,980 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,980 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,980 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:25,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:25,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:25,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:25,981 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,981 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:25,981 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:25,981 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:25,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:25,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:25,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:25,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,983 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:25,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749df9e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:25,985 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:25,985 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,986 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:25,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,991 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:25,994 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:25,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:25,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:25,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:25,994 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:25,995 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,995 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:25,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,995 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:25,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,999 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:25,999 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:25,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:25,999 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:25,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:25,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:26,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:25,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:25,999 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,000 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,000 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,000 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,000 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,000 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,001 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:26,001 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:26,101 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:26,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:26,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:26,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:26,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:26,104 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:26,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:26,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:26,105 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,105 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,106 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,107 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,107 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:26,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:26,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:26,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:26,112 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:26,112 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,112 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:26,115 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,115 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:26,115 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:26,115 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:26,117 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,117 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,118 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,118 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6201c334[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:26,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:26,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,122 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:26,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:26,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:26,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:26,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507786119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,123 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,123 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:26,125 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:26,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:26,130 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:26,130 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:26,130 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:26,130 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:26,130 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:26,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:26,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507786123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,132 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,132 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,132 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:26,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507786126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507786128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507786129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,139 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:26,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:26,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:26,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:26,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:26,140 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:26,140 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:26,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,140 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,140 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,140 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,140 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,140 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/083b5d3c4f1a4c1399f8258f743c76da 2023-03-22 17:55:26,240 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:26,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:26,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:26,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:26,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:26,243 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:26,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:26,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:26,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,245 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,245 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,245 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:26,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:26,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:26,247 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,247 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:26,247 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:26,247 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:26,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:26,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:26,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43b20a5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:26,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:26,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,251 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:26,252 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,252 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:26,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:26,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:26,256 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:26,256 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:26,256 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:26,256 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:26,256 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:26,257 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,257 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,257 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,260 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:26,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,262 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:26,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:26,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,263 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,263 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,263 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,263 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:26,263 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:26,363 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:26,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:26,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:26,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:26,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:26,366 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:26,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:26,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:26,367 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,367 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,368 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:26,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:26,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:26,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:26,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:26,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:26,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:26,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:26,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:26,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:26,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:26,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:26,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:26,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,372 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@550388f1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:26,374 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:26,374 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:26,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,376 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:26,377 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:26,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:26,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:26,379 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:26,379 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:26,379 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:26,379 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:26,379 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:26,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:26,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,387 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,387 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:26,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,387 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,387 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,388 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,388 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,388 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,388 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,388 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:26,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:26,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:26,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,388 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:26,388 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:26,488 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:26,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:26,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:26,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:26,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:26,491 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:26,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:26,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:26,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:26,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:26,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:26,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:26,495 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,495 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:26,495 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:26,495 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:26,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cff18e0[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:26,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:26,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,501 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:26,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:26,502 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:26,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:26,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:26,504 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:26,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:26,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:26,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:26,504 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:26,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:26,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,508 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:26,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,510 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:26,510 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:26,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:26,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:26,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:26,510 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:26,568 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/f3b915c14f3e4f0993f6979323704ef7 2023-03-22 17:55:26,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/5da3fd8bbb374a16a5b5b29ebd30c54a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5da3fd8bbb374a16a5b5b29ebd30c54a 2023-03-22 17:55:26,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5da3fd8bbb374a16a5b5b29ebd30c54a, entries=350, sequenceid=61, filesize=52.3 K 2023-03-22 17:55:26,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/083b5d3c4f1a4c1399f8258f743c76da as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/083b5d3c4f1a4c1399f8258f743c76da 2023-03-22 17:55:26,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/083b5d3c4f1a4c1399f8258f743c76da, entries=150, sequenceid=61, filesize=11.6 K 2023-03-22 17:55:26,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/f3b915c14f3e4f0993f6979323704ef7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f3b915c14f3e4f0993f6979323704ef7 2023-03-22 17:55:26,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f3b915c14f3e4f0993f6979323704ef7, entries=150, sequenceid=61, filesize=11.6 K 2023-03-22 17:55:26,584 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.65 KB/141980 for 2343ed3ff2707ca3be5523fce8688282 in 1689ms, sequenceid=61, compaction requested=true 2023-03-22 17:55:26,584 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:26,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:26,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:26,584 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:26,585 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:26,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:26,586 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:26,586 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/B is initiating minor compaction (all files) 2023-03-22 17:55:26,586 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/B in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,586 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/043afd621392438da094309a544f24c0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/91f7f1776bfd4b12ae2b2b8ac088b782, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/083b5d3c4f1a4c1399f8258f743c76da] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=34.8 K 2023-03-22 17:55:26,586 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:26,586 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/A is initiating minor compaction (all files) 2023-03-22 17:55:26,586 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/A in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,586 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/0b3b6df8495742ee85a266ad2e906962, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/abf29c70133f43e2a5cd42d5a5e2cd0d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5da3fd8bbb374a16a5b5b29ebd30c54a] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=102.5 K 2023-03-22 17:55:26,586 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 043afd621392438da094309a544f24c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679507722633 2023-03-22 17:55:26,586 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0b3b6df8495742ee85a266ad2e906962, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679507722633 2023-03-22 17:55:26,587 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 91f7f1776bfd4b12ae2b2b8ac088b782, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507722694 2023-03-22 17:55:26,587 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting abf29c70133f43e2a5cd42d5a5e2cd0d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507722694 2023-03-22 17:55:26,587 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 083b5d3c4f1a4c1399f8258f743c76da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507724878 2023-03-22 17:55:26,587 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5da3fd8bbb374a16a5b5b29ebd30c54a, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507724875 2023-03-22 17:55:26,604 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#B#compaction#309 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:26,607 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#A#compaction#310 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:26,611 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:26,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:26,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:26,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:26,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:26,624 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:26,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:26,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:26,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:26,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:26,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:26,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:26,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:26,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:26,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:26,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:26,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:26,632 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=138.65 KB heapSize=390.45 KB 2023-03-22 17:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:26,632 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:26,645 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303225ca7c6d3c39e49feaf973e0861d099c7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303225ca7c6d3c39e49feaf973e0861d099c7 2023-03-22 17:55:27,031 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/0a417a4d20064d22908857c545fc8cb1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0a417a4d20064d22908857c545fc8cb1 2023-03-22 17:55:27,036 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/B of 2343ed3ff2707ca3be5523fce8688282 into 0a417a4d20064d22908857c545fc8cb1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:27,036 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:27,036 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/B, priority=13, startTime=1679507726584; duration=0sec 2023-03-22 17:55:27,036 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:27,036 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:27,038 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:27,038 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/C is initiating minor compaction (all files) 2023-03-22 17:55:27,038 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/C in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:27,038 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d3d0ee4992d846e9bd926a689697a5b3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/6f6d6b193e894d31b229d7a6d615dfc0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f3b915c14f3e4f0993f6979323704ef7] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=34.8 K 2023-03-22 17:55:27,038 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d3d0ee4992d846e9bd926a689697a5b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679507722633 2023-03-22 17:55:27,039 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6f6d6b193e894d31b229d7a6d615dfc0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507722694 2023-03-22 17:55:27,039 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f3b915c14f3e4f0993f6979323704ef7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507724878 2023-03-22 17:55:27,044 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#C#compaction#312 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:27,053 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/a49d3ec542d2489b86761a0adda1dd1d 2023-03-22 17:55:27,068 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/4492214b7dc447f6a9ee3a708ab85d37 2023-03-22 17:55:27,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:27,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:27,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507787141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507787143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507787145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507787145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507787147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507787248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507787254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507787254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507787256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,452 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/54a086e85a1740ab88efd79daa00293a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/54a086e85a1740ab88efd79daa00293a 2023-03-22 17:55:27,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507787454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,459 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/C of 2343ed3ff2707ca3be5523fce8688282 into 54a086e85a1740ab88efd79daa00293a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:27,460 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:27,460 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/C, priority=13, startTime=1679507726584; duration=0sec 2023-03-22 17:55:27,460 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:27,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507787459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507787460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507787464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,477 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/1b6178afd83845a092922cf21da7442d 2023-03-22 17:55:27,480 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/a49d3ec542d2489b86761a0adda1dd1d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a49d3ec542d2489b86761a0adda1dd1d 2023-03-22 17:55:27,483 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a49d3ec542d2489b86761a0adda1dd1d, entries=150, sequenceid=97, filesize=25.1 K 2023-03-22 17:55:27,484 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/4492214b7dc447f6a9ee3a708ab85d37 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/4492214b7dc447f6a9ee3a708ab85d37 2023-03-22 17:55:27,488 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/4492214b7dc447f6a9ee3a708ab85d37, entries=150, sequenceid=97, filesize=11.6 K 2023-03-22 17:55:27,488 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/1b6178afd83845a092922cf21da7442d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1b6178afd83845a092922cf21da7442d 2023-03-22 17:55:27,491 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1b6178afd83845a092922cf21da7442d, entries=150, sequenceid=97, filesize=11.6 K 2023-03-22 17:55:27,492 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~390.41 KB/399776, currentSize=46.96 KB/48090 for 2343ed3ff2707ca3be5523fce8688282 in 860ms, sequenceid=97, compaction requested=false 2023-03-22 17:55:27,492 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:27,492 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:27,494 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:27,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@700b6028[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:27,495 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:27,496 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:27,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:27,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:27,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:27,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:27,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:27,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:27,500 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:27,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:27,503 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:27,503 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:27,503 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:27,503 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:27,503 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:27,505 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,505 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:27,505 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:27,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:27,505 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:27,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:27,508 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:27,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:27,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:27,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:27,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:27,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:27,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:27,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:27,509 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:27,509 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:27,509 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:27,509 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:27,509 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:27,509 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,609 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:27,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:27,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:27,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:27,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:27,613 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:27,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:27,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:27,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:27,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:27,614 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:27,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:27,614 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:27,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:27,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:27,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:27,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:27,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:27,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:27,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:27,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:27,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:27,616 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:27,616 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:27,616 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:27,616 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:27,616 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:27,616 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:27,617 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:27,617 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:27,617 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:27,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:27,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507727785, max timestamp=1679507727792Num uniques 859; 2023-03-22 17:55:27,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:27,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507727785, max timestamp=1679507727792Num uniques 859; 2023-03-22 17:55:27,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:27,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507727785, max timestamp=1679507727792Num uniques 859; 2023-03-22 17:55:27,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:27,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507787829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507787830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507787831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507787834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,842 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/e73ed84e4523404b97f43adf20b62be3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e73ed84e4523404b97f43adf20b62be3 2023-03-22 17:55:27,846 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/A of 2343ed3ff2707ca3be5523fce8688282 into e73ed84e4523404b97f43adf20b62be3(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-22 17:55:27,846 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:27,846 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/A, priority=13, startTime=1679507726584; duration=1sec 2023-03-22 17:55:27,846 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:27,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507787939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507787939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507787939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:27,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:27,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507787939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,032 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303229d879fd52b1e47ebb3730b6402c57a28 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303229d879fd52b1e47ebb3730b6402c57a28 2023-03-22 17:55:28,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507788145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507788145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507788145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507788146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,437 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/b01a853384524d3f9bc86562737a3ecb 2023-03-22 17:55:28,450 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/8fe4ff785f8d4a31b02a699f21a8809c 2023-03-22 17:55:28,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507788447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507788447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507788448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507788459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,466 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/125705772b5f439980dc6fdc92a3fa1a 2023-03-22 17:55:28,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/b01a853384524d3f9bc86562737a3ecb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b01a853384524d3f9bc86562737a3ecb 2023-03-22 17:55:28,475 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b01a853384524d3f9bc86562737a3ecb, entries=150, sequenceid=109, filesize=25.1 K 2023-03-22 17:55:28,476 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/8fe4ff785f8d4a31b02a699f21a8809c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/8fe4ff785f8d4a31b02a699f21a8809c 2023-03-22 17:55:28,480 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/8fe4ff785f8d4a31b02a699f21a8809c, entries=150, sequenceid=109, filesize=11.6 K 2023-03-22 17:55:28,481 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/125705772b5f439980dc6fdc92a3fa1a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/125705772b5f439980dc6fdc92a3fa1a 2023-03-22 17:55:28,484 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/125705772b5f439980dc6fdc92a3fa1a, entries=150, sequenceid=109, filesize=11.6 K 2023-03-22 17:55:28,490 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2343ed3ff2707ca3be5523fce8688282 in 874ms, sequenceid=109, compaction requested=true 2023-03-22 17:55:28,490 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:28,490 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:28,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:28,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:28,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d684884[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:28,495 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:28,495 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:28,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:28,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:28,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:28,497 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,497 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:28,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:28,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:28,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:28,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:28,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:28,502 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:28,502 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:28,502 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:28,502 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:28,502 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:28,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:28,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,505 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:28,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:28,507 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,514 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:28,514 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:28,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:28,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:28,514 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:28,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:28,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:28,514 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:28,515 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:28,515 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:28,514 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:28,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:28,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:28,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:28,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,615 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:28,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:28,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:28,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:28,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:28,618 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:28,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:28,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:28,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:28,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:28,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:28,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:28,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:28,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:28,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:28,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:28,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:28,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:28,622 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:28,622 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:28,622 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:28,623 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:28,623 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:28,623 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:28,623 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:28,623 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-22 17:55:28,623 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:28,623 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:28,623 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:28,623 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:28,623 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:28,623 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:28,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:28,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:28,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507788966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507788967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507788967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:28,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:28,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507788968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,043 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322cc07dbb9551743e9bba4853407df7123 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322cc07dbb9551743e9bba4853407df7123 2023-03-22 17:55:29,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507789075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507789075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507789075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507789162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,170 DEBUG [Thread-2385] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:29,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507789281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507789281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507789281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,453 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/274ad8987591402d8a0ae5aa8f340e90 2023-03-22 17:55:29,463 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/03ee4005700a40acaf7f209b1d9dee4f 2023-03-22 17:55:29,473 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/93c5db0af66948b09241a3ee20705f4e 2023-03-22 17:55:29,476 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/274ad8987591402d8a0ae5aa8f340e90 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/274ad8987591402d8a0ae5aa8f340e90 2023-03-22 17:55:29,479 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/274ad8987591402d8a0ae5aa8f340e90, entries=150, sequenceid=147, filesize=25.3 K 2023-03-22 17:55:29,480 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/03ee4005700a40acaf7f209b1d9dee4f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/03ee4005700a40acaf7f209b1d9dee4f 2023-03-22 17:55:29,483 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/03ee4005700a40acaf7f209b1d9dee4f, entries=150, sequenceid=147, filesize=11.8 K 2023-03-22 17:55:29,484 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/93c5db0af66948b09241a3ee20705f4e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/93c5db0af66948b09241a3ee20705f4e 2023-03-22 17:55:29,487 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/93c5db0af66948b09241a3ee20705f4e, entries=150, sequenceid=147, filesize=11.8 K 2023-03-22 17:55:29,487 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-03-22 17:55:29,487 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2343ed3ff2707ca3be5523fce8688282 in 864ms, sequenceid=147, compaction requested=true 2023-03-22 17:55:29,487 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:29,487 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:29,489 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646519d8[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:29,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:29,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,494 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:29,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:29,497 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:29,497 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:29,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:29,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:29,498 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:29,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,499 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:29,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,500 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:29,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:29,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:29,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,504 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:29,504 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:29,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:29,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:55:29,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:29,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:29,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:29,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:29,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:29,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:29,605 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:29,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:29,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:29,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:29,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:29,608 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:29,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:29,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:29,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:29,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,610 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:29,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:29,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:29,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:29,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:29,612 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,612 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:29,612 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:29,612 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:29,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:29,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b925ac[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:29,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:29,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:29,627 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:29,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:29,629 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:29,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:29,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:29,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:29,629 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:29,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,633 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:29,636 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:29,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507729616, max timestamp=1679507729634Num uniques 859; 2023-03-22 17:55:29,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:29,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507729616, max timestamp=1679507729634Num uniques 859; 2023-03-22 17:55:29,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507729616, max timestamp=1679507729634Num uniques 859; 2023-03-22 17:55:29,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507789628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,637 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:29,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:29,637 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:29,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:29,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:29,637 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:29,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:29,637 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:29,637 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:29,637 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:29,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:29,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507789682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507789692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,737 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:29,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:29,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:29,741 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:29,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:29,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:29,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:29,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:29,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:29,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:29,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:29,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:29,745 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,745 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:29,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507789738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,746 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:29,746 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b180fb3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:29,753 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:29,753 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:29,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:29,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:29,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:29,758 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:29,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:29,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:29,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:29,758 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:29,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:29,761 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,762 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:29,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:29,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:29,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:29,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:29,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:29,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:29,768 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:29,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,768 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:29,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:29,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507789793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507789799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,868 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:29,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:29,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:29,872 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:29,872 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:29,872 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:29,872 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:29,872 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,872 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:29,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:29,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:29,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:29,876 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,876 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:29,876 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:29,876 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:29,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44db5e3c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:29,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:29,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,881 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:29,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:29,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:29,885 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:29,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:29,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:29,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:29,885 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:29,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:29,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:29,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:29,887 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,888 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,890 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,890 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:29,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:29,890 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:29,890 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:29,890 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:29,890 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:29,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:29,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:29,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:29,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:29,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:29,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507789948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:29,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507789976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,991 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:29,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:29,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:29,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:29,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:29,994 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:29,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:29,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:29,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:29,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,996 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:29,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:29,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:29,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:29,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:29,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:29,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:29,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:29,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:30,000 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:30,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8bc24a6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:30,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:30,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,003 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322a23a2b306de84f6992ba69835843eb1a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a23a2b306de84f6992ba69835843eb1a 2023-03-22 17:55:30,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:30,004 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507790000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:30,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:30,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:30,004 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,005 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:30,006 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:30,008 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:30,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:30,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:30,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:30,009 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:30,010 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/82c75fb4ca1e406b992672eaacf6a3c6 2023-03-22 17:55:30,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507790005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:30,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:30,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,016 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:30,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,016 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:30,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:30,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/b3b6cd328aa34c099c1abf7087439bcd 2023-03-22 17:55:30,116 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:30,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:30,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:30,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:30,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:30,119 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:30,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:30,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:30,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,121 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:30,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:30,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:30,122 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:30,122 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,122 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:30,123 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,123 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:30,123 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:30,123 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:30,124 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,124 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:30,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e1e458[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:30,126 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:30,126 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,128 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:30,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:30,129 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:30,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:30,132 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:30,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:30,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:30,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:30,132 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:30,133 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,133 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,133 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,134 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:30,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:30,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:30,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,137 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,137 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:30,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,238 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:30,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:30,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:30,241 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:30,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:30,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:30,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,248 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,248 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:30,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:30,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:30,250 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,250 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:30,250 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:30,250 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:30,259 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507790254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:30,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aef1e40[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:30,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:30,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:30,262 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,262 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:30,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:30,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:30,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:30,264 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:30,266 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:30,266 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:30,266 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:30,266 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:30,266 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:30,269 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,269 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,269 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,270 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,270 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:30,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,273 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:30,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,273 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:30,273 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507790306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:30,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507790314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,373 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:30,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:30,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:30,378 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:30,378 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,378 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:30,378 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:30,378 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,378 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,379 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,379 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,380 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,380 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:30,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:30,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:30,381 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:30,381 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,381 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:30,382 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:30,382 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:30,383 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,383 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:30,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@345b8f9f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:30,386 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:30,386 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,388 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:30,392 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,392 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,393 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:30,395 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:30,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:30,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:30,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:30,396 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:30,397 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,397 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,397 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,398 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,401 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,400 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:30,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:30,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,401 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,401 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:30,401 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:30,401 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,401 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,401 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,401 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:30,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/a234ad8f33674d92b1032d8561a7bebd 2023-03-22 17:55:30,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/82c75fb4ca1e406b992672eaacf6a3c6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/82c75fb4ca1e406b992672eaacf6a3c6 2023-03-22 17:55:30,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/82c75fb4ca1e406b992672eaacf6a3c6, entries=200, sequenceid=158, filesize=32.1 K 2023-03-22 17:55:30,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/b3b6cd328aa34c099c1abf7087439bcd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b3b6cd328aa34c099c1abf7087439bcd 2023-03-22 17:55:30,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b3b6cd328aa34c099c1abf7087439bcd, entries=150, sequenceid=158, filesize=11.8 K 2023-03-22 17:55:30,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/a234ad8f33674d92b1032d8561a7bebd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/a234ad8f33674d92b1032d8561a7bebd 2023-03-22 17:55:30,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/a234ad8f33674d92b1032d8561a7bebd, entries=150, sequenceid=158, filesize=11.8 K 2023-03-22 17:55:30,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2343ed3ff2707ca3be5523fce8688282 in 862ms, sequenceid=158, compaction requested=true 2023-03-22 17:55:30,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:30,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:30,451 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:55:30,451 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:55:30,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:30,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:30,453 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136123 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:55:30,453 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/A is initiating minor compaction (all files) 2023-03-22 17:55:30,453 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/A in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,453 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e73ed84e4523404b97f43adf20b62be3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a49d3ec542d2489b86761a0adda1dd1d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b01a853384524d3f9bc86562737a3ecb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/274ad8987591402d8a0ae5aa8f340e90, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/82c75fb4ca1e406b992672eaacf6a3c6] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=132.9 K 2023-03-22 17:55:30,453 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:55:30,453 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/B is initiating minor compaction (all files) 2023-03-22 17:55:30,453 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/B in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,453 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0a417a4d20064d22908857c545fc8cb1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/4492214b7dc447f6a9ee3a708ab85d37, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/8fe4ff785f8d4a31b02a699f21a8809c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/03ee4005700a40acaf7f209b1d9dee4f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b3b6cd328aa34c099c1abf7087439bcd] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=58.4 K 2023-03-22 17:55:30,454 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e73ed84e4523404b97f43adf20b62be3, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507724878 2023-03-22 17:55:30,454 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0a417a4d20064d22908857c545fc8cb1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507724878 2023-03-22 17:55:30,454 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a49d3ec542d2489b86761a0adda1dd1d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507724994 2023-03-22 17:55:30,454 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4492214b7dc447f6a9ee3a708ab85d37, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507724994 2023-03-22 17:55:30,455 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b01a853384524d3f9bc86562737a3ecb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679507727144 2023-03-22 17:55:30,455 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 8fe4ff785f8d4a31b02a699f21a8809c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679507727144 2023-03-22 17:55:30,455 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 274ad8987591402d8a0ae5aa8f340e90, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679507727829 2023-03-22 17:55:30,455 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 03ee4005700a40acaf7f209b1d9dee4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679507727829 2023-03-22 17:55:30,455 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 82c75fb4ca1e406b992672eaacf6a3c6, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507728965 2023-03-22 17:55:30,455 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b3b6cd328aa34c099c1abf7087439bcd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507728966 2023-03-22 17:55:30,473 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#B#compaction#325 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:30,476 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#A#compaction#324 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:30,488 WARN [DataStreamer for file /user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303227b300053267b4caba9fe40fda09bef69_del] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-22 17:55:30,501 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:30,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:30,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:30,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:30,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:30,512 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:30,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:30,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:30,513 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,513 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,520 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:30,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:30,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:30,521 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:30,521 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,521 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:30,521 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:30,522 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:30,522 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,522 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:30,522 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-22 17:55:30,522 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:30,522 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:30,524 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/1a6836d282c94bd7861cd3f688fc16a6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/1a6836d282c94bd7861cd3f688fc16a6 2023-03-22 17:55:30,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:30,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:30,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:30,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:30,528 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/A of 2343ed3ff2707ca3be5523fce8688282 into 1a6836d282c94bd7861cd3f688fc16a6(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:30,528 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:30,528 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/A, priority=11, startTime=1679507730451; duration=0sec 2023-03-22 17:55:30,528 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:30,529 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:55:30,530 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:55:30,530 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/C is initiating minor compaction (all files) 2023-03-22 17:55:30,530 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/C in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,530 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/54a086e85a1740ab88efd79daa00293a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1b6178afd83845a092922cf21da7442d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/125705772b5f439980dc6fdc92a3fa1a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/93c5db0af66948b09241a3ee20705f4e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/a234ad8f33674d92b1032d8561a7bebd] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=58.4 K 2023-03-22 17:55:30,532 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 54a086e85a1740ab88efd79daa00293a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507724878 2023-03-22 17:55:30,532 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1b6178afd83845a092922cf21da7442d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679507724994 2023-03-22 17:55:30,532 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 125705772b5f439980dc6fdc92a3fa1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679507727144 2023-03-22 17:55:30,533 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 93c5db0af66948b09241a3ee20705f4e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679507727829 2023-03-22 17:55:30,533 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a234ad8f33674d92b1032d8561a7bebd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507728966 2023-03-22 17:55:30,546 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#C#compaction#327 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:30,552 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322308b8414d6c74b43a59169ba1dd50e08 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322308b8414d6c74b43a59169ba1dd50e08 2023-03-22 17:55:30,564 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/72ed888691da42e09320b353a36fc6af 2023-03-22 17:55:30,584 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/5695858e752f4874acca3612f01d7f1f 2023-03-22 17:55:30,598 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/4b8f6d6f79684e639353462747f95a42 2023-03-22 17:55:30,602 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/72ed888691da42e09320b353a36fc6af as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/72ed888691da42e09320b353a36fc6af 2023-03-22 17:55:30,605 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/72ed888691da42e09320b353a36fc6af, entries=150, sequenceid=193, filesize=25.3 K 2023-03-22 17:55:30,606 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/5695858e752f4874acca3612f01d7f1f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/5695858e752f4874acca3612f01d7f1f 2023-03-22 17:55:30,609 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/5695858e752f4874acca3612f01d7f1f, entries=150, sequenceid=193, filesize=11.8 K 2023-03-22 17:55:30,609 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/4b8f6d6f79684e639353462747f95a42 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b8f6d6f79684e639353462747f95a42 2023-03-22 17:55:30,613 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b8f6d6f79684e639353462747f95a42, entries=150, sequenceid=193, filesize=11.8 K 2023-03-22 17:55:30,614 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 2343ed3ff2707ca3be5523fce8688282 in 92ms, sequenceid=193, compaction requested=false 2023-03-22 17:55:30,614 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:30,614 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:30,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:30,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:30,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:30,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:30,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dab14a6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:30,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:30,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:30,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:30,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:30,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:30,621 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:30,621 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,621 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,621 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:30,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:30,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:30,624 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:30,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:30,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:30,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:30,624 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:30,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,627 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:30,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:30,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:30,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,631 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:30,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,631 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:30,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,631 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,731 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:30,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:30,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:30,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:30,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:30,735 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:30,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:30,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:30,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,737 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:30,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:30,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:30,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:30,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:30,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:30,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:30,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:30,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:30,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:30,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:30,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:30,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:30,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1326e86f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:30,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:30,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,751 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,752 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:30,752 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:30,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:30,752 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:30,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:30,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:30,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:30,756 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:30,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:30,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:30,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:30,756 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:30,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,758 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,758 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:30,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:30,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,764 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:30,764 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:30,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:30,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:30,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:30,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:30,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:30,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:30,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:30,864 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:30,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:30,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:30,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:30,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:30,867 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:30,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:30,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:30,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,867 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,869 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,869 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,869 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:30,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:30,870 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,870 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:30,870 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:30,870 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:30,872 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4169f955[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:30,873 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:30,873 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,874 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:30,874 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:30,874 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:30,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:30,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:30,879 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:30,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:30,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:30,879 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:30,879 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:30,880 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,880 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,880 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,884 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:30,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:30,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:30,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:30,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:30,884 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:30,884 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:30,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,885 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,892 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/1c9f05de8aef42ce9a555b3623714df4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/1c9f05de8aef42ce9a555b3623714df4 2023-03-22 17:55:30,896 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/B of 2343ed3ff2707ca3be5523fce8688282 into 1c9f05de8aef42ce9a555b3623714df4(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:30,896 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:30,896 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/B, priority=11, startTime=1679507730451; duration=0sec 2023-03-22 17:55:30,896 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:30,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507730874, max timestamp=1679507730901Num uniques 859; 2023-03-22 17:55:30,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:55:30,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507730874, max timestamp=1679507730901Num uniques 859; 2023-03-22 17:55:30,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:55:30,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507730874, max timestamp=1679507730901Num uniques 859; 2023-03-22 17:55:30,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:55:30,958 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/8a951bef60294f73b1ade66038ec8671 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/8a951bef60294f73b1ade66038ec8671 2023-03-22 17:55:30,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:30,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507790949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:30,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507790957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,963 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/C of 2343ed3ff2707ca3be5523fce8688282 into 8a951bef60294f73b1ade66038ec8671(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:30,963 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:30,963 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/C, priority=11, startTime=1679507730451; duration=0sec 2023-03-22 17:55:30,963 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:30,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:30,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507790958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,984 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:30,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:30,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:30,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:30,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:30,987 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:30,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:30,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:30,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:30,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:30,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:30,990 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:30,991 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,991 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,991 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:30,991 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:30,991 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:30,991 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:30,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:30,994 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2926b190[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:30,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:30,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,998 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:30,998 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:30,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:31,000 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:31,003 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:31,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:31,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:31,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:31,003 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:31,004 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,004 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,004 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,005 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:31,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,014 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:31,014 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,014 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507791060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507791064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507791067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,114 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:31,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:31,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:31,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:31,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:31,117 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:31,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:31,117 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:31,118 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,118 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:31,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:31,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:31,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:31,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:31,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:31,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:31,121 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,121 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:31,121 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:31,121 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:31,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:31,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,122 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:31,122 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:31,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:31,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:31,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cad8e36[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:31,125 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:31,125 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,126 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:31,126 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,126 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:31,128 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:31,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:31,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:31,130 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:31,130 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:31,130 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:31,130 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:31,130 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:31,134 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,134 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,134 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:31,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:31,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:31,138 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,138 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:31,138 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,139 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:31,239 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:31,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:31,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:31,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:31,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:31,242 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:31,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:31,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:31,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:31,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:31,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:31,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:31,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:31,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:31,245 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,245 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:31,245 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:31,245 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:31,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:31,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:31,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:31,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:31,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:31,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86c7e5c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:31,251 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,251 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:31,251 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:31,251 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:31,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,251 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:31,253 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:31,253 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,253 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,253 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:31,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:31,255 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:31,255 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:31,255 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:31,255 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:31,255 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:31,256 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,256 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,256 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,257 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:31,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:31,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,260 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,260 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:31,261 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:31,260 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,264 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032212e3888d041c49ae8953202c1902f244 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032212e3888d041c49ae8953202c1902f244 2023-03-22 17:55:31,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507791267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507791272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507791276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,362 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:31,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:31,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:31,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:31,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:31,384 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:31,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:31,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:31,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,387 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,387 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:31,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:31,406 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,406 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:31,406 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:31,406 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:31,410 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,411 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:31,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6419d55[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:31,413 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:31,413 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,414 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,414 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,414 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:31,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:31,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:31,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:31,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:31,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:31,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:31,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:31,420 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:31,420 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:31,420 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:31,420 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:31,420 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:31,424 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,426 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,426 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:31,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,430 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,430 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:31,430 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:31,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:31,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,530 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:31,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:31,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:31,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:31,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:31,534 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:31,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:31,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:31,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,535 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,535 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,535 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,536 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:31,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:31,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:31,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:31,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:31,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:31,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:31,539 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a77063c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:31,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:31,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:31,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:31,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,543 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:31,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:31,544 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:31,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:31,546 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:31,546 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:31,546 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:31,546 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:31,546 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:31,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:31,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:31,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,555 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:31,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,555 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:31,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507791577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507791584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507791591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,655 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:31,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:31,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:31,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:31,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:31,658 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:31,658 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,658 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:31,658 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:31,659 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,659 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,660 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:31,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:31,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:31,662 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,662 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:31,662 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:31,662 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:31,664 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aebf77a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:31,665 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:31,665 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,666 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,667 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:31,667 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:31,667 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,667 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:31,667 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:31,669 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=205, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/e287a64c6c5b456694c891409220ffb8 2023-03-22 17:55:31,669 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:31,669 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,669 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:31,669 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:31,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:31,672 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:31,672 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:31,672 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:31,672 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:31,672 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:31,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,675 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,676 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:31,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:31,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,680 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:31,680 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:31,780 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:31,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:31,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:31,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:31,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:31,783 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:31,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:31,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:31,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,785 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:31,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:31,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:31,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:31,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:31,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:31,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:31,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:31,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:31,790 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,790 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,790 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:31,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@319eece5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:31,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:31,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:31,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:31,793 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:31,792 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,793 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:31,795 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,795 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:31,795 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,795 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:31,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:31,797 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:31,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:31,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:31,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:31,797 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,799 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,800 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,802 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:31,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:31,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:31,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,803 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:31,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,803 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:31,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,903 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:31,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:31,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:31,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:31,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:31,906 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:31,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:31,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:31,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,908 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,908 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,908 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:31,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:31,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:31,909 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,910 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:31,910 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:31,910 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:31,912 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:31,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:31,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b3f74b6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:31,914 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:31,914 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:31,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:31,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:31,915 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,915 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:31,917 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:31,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,917 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:31,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:31,920 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:31,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:31,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:31,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:31,920 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:31,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:31,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,922 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:31,923 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:31,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:31,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:31,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:31,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:31,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:31,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:31,929 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:31,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:31,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:31,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:31,929 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:31,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:31,930 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:31,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:31,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:31,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507791991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:31,996 DEBUG [Thread-2383] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:32,030 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:32,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:32,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:32,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:32,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:32,033 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:32,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:32,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:32,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,034 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,034 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:32,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:32,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:32,036 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,036 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:32,036 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:32,036 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:32,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,039 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195bd30b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:32,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:32,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,040 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:32,041 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:32,041 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:32,041 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,041 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:32,043 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:32,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:32,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:32,045 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:32,045 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:32,045 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:32,045 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:32,045 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:32,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:32,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,050 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,050 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,050 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,050 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,050 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:32,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:32,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:32,050 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:32,050 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:32,050 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:32,050 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,051 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,051 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:32,051 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:32,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/24e9cc124d924cf8aac4aeeddb423280 2023-03-22 17:55:32,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:32,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507792082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:32,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507792094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507792100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,151 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:32,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:32,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:32,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:32,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:32,154 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:32,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:32,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:32,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,156 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:32,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:32,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:32,157 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,157 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:32,157 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:32,157 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:32,157 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:32,159 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,159 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:32,161 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:32,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c8ea77d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:32,161 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,162 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:32,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:32,164 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:32,164 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:32,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:32,166 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:32,166 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:32,166 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:32,166 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:32,166 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:32,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,169 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,177 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:32,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:32,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:32,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:32,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:32,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:32,177 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,178 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:32,178 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:32,178 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,178 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,178 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,278 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:32,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:32,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:32,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:32,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:32,281 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:32,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:32,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:32,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,283 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:32,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:32,285 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,285 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:32,285 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:32,285 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:32,286 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,287 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab189cd[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:32,288 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:32,288 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,289 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:32,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:32,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:32,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,292 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:32,294 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:32,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:32,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:32,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:32,294 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:32,296 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,296 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,296 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:32,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:32,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,299 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,301 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:32,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:32,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:32,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:32,301 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:32,301 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:32,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:32,401 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:32,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:32,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:32,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:32,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:32,405 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:32,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:32,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:32,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,408 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,408 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:32,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:32,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:32,410 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,410 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:32,410 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:32,410 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:32,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:32,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,415 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:32,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aebdce9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:32,417 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:32,417 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,418 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,418 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:32,418 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:32,418 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:32,418 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,419 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:32,421 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:32,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:32,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:32,423 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:32,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:32,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:32,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:32,423 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:32,425 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,425 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,425 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:32,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:32,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,434 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,434 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:32,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,434 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,435 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,435 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,435 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,435 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,435 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:32,435 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:32,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:32,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:32,434 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:32,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:32,435 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:32,436 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:32,489 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/4ac7b030e5ee46d3aed4c78e2c5ac5f7 2023-03-22 17:55:32,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/e287a64c6c5b456694c891409220ffb8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e287a64c6c5b456694c891409220ffb8 2023-03-22 17:55:32,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e287a64c6c5b456694c891409220ffb8, entries=350, sequenceid=205, filesize=52.6 K 2023-03-22 17:55:32,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/24e9cc124d924cf8aac4aeeddb423280 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/24e9cc124d924cf8aac4aeeddb423280 2023-03-22 17:55:32,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/24e9cc124d924cf8aac4aeeddb423280, entries=150, sequenceid=205, filesize=11.8 K 2023-03-22 17:55:32,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/4ac7b030e5ee46d3aed4c78e2c5ac5f7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4ac7b030e5ee46d3aed4c78e2c5ac5f7 2023-03-22 17:55:32,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4ac7b030e5ee46d3aed4c78e2c5ac5f7, entries=150, sequenceid=205, filesize=11.8 K 2023-03-22 17:55:32,509 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 2343ed3ff2707ca3be5523fce8688282 in 1658ms, sequenceid=205, compaction requested=true 2023-03-22 17:55:32,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:32,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:32,509 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:32,509 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:32,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:32,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:32,510 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105926 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:32,510 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/A is initiating minor compaction (all files) 2023-03-22 17:55:32,510 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/A in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,510 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/1a6836d282c94bd7861cd3f688fc16a6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/72ed888691da42e09320b353a36fc6af, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e287a64c6c5b456694c891409220ffb8] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=103.4 K 2023-03-22 17:55:32,511 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:32,511 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/B is initiating minor compaction (all files) 2023-03-22 17:55:32,511 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/B in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,511 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/1c9f05de8aef42ce9a555b3623714df4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/5695858e752f4874acca3612f01d7f1f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/24e9cc124d924cf8aac4aeeddb423280] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=35.5 K 2023-03-22 17:55:32,511 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1a6836d282c94bd7861cd3f688fc16a6, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507728966 2023-03-22 17:55:32,511 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1c9f05de8aef42ce9a555b3623714df4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507728966 2023-03-22 17:55:32,511 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 72ed888691da42e09320b353a36fc6af, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1679507729678 2023-03-22 17:55:32,512 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 5695858e752f4874acca3612f01d7f1f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1679507729678 2023-03-22 17:55:32,512 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e287a64c6c5b456694c891409220ffb8, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507730787 2023-03-22 17:55:32,512 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 24e9cc124d924cf8aac4aeeddb423280, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507730839 2023-03-22 17:55:32,525 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#B#compaction#334 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:32,536 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:32,537 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#A#compaction#333 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:32,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:32,544 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:32,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:32,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:32,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,545 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,546 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:32,546 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:32,546 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:32,546 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:32,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:32,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:32,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:32,547 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:32,547 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:32,547 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:32,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:32,548 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-22 17:55:32,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:32,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:32,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:32,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:32,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:32,549 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:32,938 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/152a63f9c8a74511a28e9d1ce96285a8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/152a63f9c8a74511a28e9d1ce96285a8 2023-03-22 17:55:32,942 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/B of 2343ed3ff2707ca3be5523fce8688282 into 152a63f9c8a74511a28e9d1ce96285a8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:32,942 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:32,942 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/B, priority=13, startTime=1679507732509; duration=0sec 2023-03-22 17:55:32,942 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:32,942 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:32,943 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:32,943 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/C is initiating minor compaction (all files) 2023-03-22 17:55:32,943 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/C in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:32,943 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/8a951bef60294f73b1ade66038ec8671, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b8f6d6f79684e639353462747f95a42, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4ac7b030e5ee46d3aed4c78e2c5ac5f7] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=35.5 K 2023-03-22 17:55:32,944 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 8a951bef60294f73b1ade66038ec8671, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507728966 2023-03-22 17:55:32,944 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4b8f6d6f79684e639353462747f95a42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1679507729678 2023-03-22 17:55:32,945 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4ac7b030e5ee46d3aed4c78e2c5ac5f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507730839 2023-03-22 17:55:32,950 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#C#compaction#336 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:32,969 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303222a1a5fee9933441e84ba7d635e648b7b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303222a1a5fee9933441e84ba7d635e648b7b 2023-03-22 17:55:33,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:33,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:33,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507793113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507793117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507793117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40258 deadline: 1679507793203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,210 DEBUG [Thread-2385] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8212 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:33,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507793224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507793225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,359 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/4b9172304f834b42b076ff5ca2531649 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b9172304f834b42b076ff5ca2531649 2023-03-22 17:55:33,363 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/C of 2343ed3ff2707ca3be5523fce8688282 into 4b9172304f834b42b076ff5ca2531649(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:33,363 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:33,363 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/C, priority=13, startTime=1679507732509; duration=0sec 2023-03-22 17:55:33,363 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:33,373 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=242, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/036eacfb09c24bcd858b43b1e7e6bc55 2023-03-22 17:55:33,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507793431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507793432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40284 deadline: 1679507793739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:33,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40226 deadline: 1679507793739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,757 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/227916012d0f4b14bed3eac56152b8bb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/227916012d0f4b14bed3eac56152b8bb 2023-03-22 17:55:33,761 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/A of 2343ed3ff2707ca3be5523fce8688282 into 227916012d0f4b14bed3eac56152b8bb(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-22 17:55:33,761 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:33,761 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/A, priority=13, startTime=1679507732509; duration=1sec 2023-03-22 17:55:33,761 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:33,794 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/d214720099d44b9bbe7a4168a980f872 2023-03-22 17:55:33,803 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/1ed2b2b35002448b95842852a12a4c02 2023-03-22 17:55:33,806 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/036eacfb09c24bcd858b43b1e7e6bc55 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/036eacfb09c24bcd858b43b1e7e6bc55 2023-03-22 17:55:33,810 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/036eacfb09c24bcd858b43b1e7e6bc55, entries=150, sequenceid=242, filesize=25.3 K 2023-03-22 17:55:33,810 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/d214720099d44b9bbe7a4168a980f872 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d214720099d44b9bbe7a4168a980f872 2023-03-22 17:55:33,814 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d214720099d44b9bbe7a4168a980f872, entries=150, sequenceid=242, filesize=11.8 K 2023-03-22 17:55:33,815 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/1ed2b2b35002448b95842852a12a4c02 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1ed2b2b35002448b95842852a12a4c02 2023-03-22 17:55:33,819 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1ed2b2b35002448b95842852a12a4c02, entries=150, sequenceid=242, filesize=11.8 K 2023-03-22 17:55:33,820 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 2343ed3ff2707ca3be5523fce8688282 in 1272ms, sequenceid=242, compaction requested=false 2023-03-22 17:55:33,821 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:33,821 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:33,823 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,823 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:33,823 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:33,823 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5def8734[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:33,826 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:33,826 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:33,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:33,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:33,828 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:33,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:33,830 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:33,830 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:33,832 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:33,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:33,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:33,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:33,833 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:33,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:33,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:33,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:33,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:33,836 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:33,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:33,839 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:33,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:33,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:33,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:33,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:33,839 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:33,839 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:33,939 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:33,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:33,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:33,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:33,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:33,943 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:33,943 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:33,943 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:33,943 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:33,943 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,943 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:33,944 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:33,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:33,945 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:33,945 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:33,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:33,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:33,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:33,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:33,946 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:33,946 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:33,946 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:33,946 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:33,946 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:33,946 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:33,946 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:33,946 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:33,947 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:33,947 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:33,947 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:33,947 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:33,947 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:33,947 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:34,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:34,248 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:34,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507734280, max timestamp=1679507734294Num uniques 859; 2023-03-22 17:55:34,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507734280, max timestamp=1679507734294Num uniques 859; 2023-03-22 17:55:34,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:34,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507734280, max timestamp=1679507734294Num uniques 859; 2023-03-22 17:55:34,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:34,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:34,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:34,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507794347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:34,361 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303223236e93d240d459480d49501450c9b22 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223236e93d240d459480d49501450c9b22 2023-03-22 17:55:34,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507794355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:34,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:34,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507794456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:34,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:34,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507794465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:34,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:34,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507794660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:34,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:34,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507794670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:34,768 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=255, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/9586133263a24923b63312a3abb77aa2 2023-03-22 17:55:34,779 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/7cbc0ea7d73b4388a5305584bc2f3656 2023-03-22 17:55:34,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:34,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507794968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:34,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:34,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507794976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:35,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507795127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,133 DEBUG [Thread-2381] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:35,191 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/41517e3c24b14536b1683a71426422ae 2023-03-22 17:55:35,196 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/9586133263a24923b63312a3abb77aa2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9586133263a24923b63312a3abb77aa2 2023-03-22 17:55:35,201 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9586133263a24923b63312a3abb77aa2, entries=150, sequenceid=255, filesize=25.3 K 2023-03-22 17:55:35,203 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/7cbc0ea7d73b4388a5305584bc2f3656 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7cbc0ea7d73b4388a5305584bc2f3656 2023-03-22 17:55:35,208 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7cbc0ea7d73b4388a5305584bc2f3656, entries=150, sequenceid=255, filesize=11.8 K 2023-03-22 17:55:35,209 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/41517e3c24b14536b1683a71426422ae as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/41517e3c24b14536b1683a71426422ae 2023-03-22 17:55:35,214 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/41517e3c24b14536b1683a71426422ae, entries=150, sequenceid=255, filesize=11.8 K 2023-03-22 17:55:35,215 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2343ed3ff2707ca3be5523fce8688282 in 1269ms, sequenceid=255, compaction requested=true 2023-03-22 17:55:35,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:35,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:35,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:35,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:35,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34b376bb[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:35,220 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:35,221 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:35,222 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:35,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:35,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:35,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:35,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:35,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:35,224 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:35,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:35,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:35,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:35,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:35,227 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:35,227 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:35,227 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:35,227 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:35,227 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:35,229 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,229 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:35,229 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:35,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,239 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:35,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:35,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:35,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:35,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:35,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:35,246 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:35,246 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:35,346 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:35,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:35,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:35,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:35,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:35,349 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:35,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:35,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:35,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:35,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:35,351 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:35,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:35,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:35,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:35,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:35,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:35,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:35,353 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,353 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:35,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:35,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:35,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:35,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:35,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:35,353 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:35,353 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:35,353 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-22 17:55:35,354 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:35,354 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:35,354 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:35,354 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:35,354 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:35,354 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:35,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:35,480 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:35,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:35,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507795512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:35,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507795512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:35,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507795620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:35,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507795620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,773 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322de21c052aa0540c7a152d873def560dd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322de21c052aa0540c7a152d873def560dd 2023-03-22 17:55:35,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:35,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507795827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:35,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:35,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507795827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:36,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40250 deadline: 1679507796032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,039 DEBUG [Thread-2383] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8208 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:36,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:36,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507796135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:36,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507796135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,179 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=291, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/3e749f660de744d1b6cc71cee50c76c2 2023-03-22 17:55:36,201 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/76f081d212104588b46195af36a0525d 2023-03-22 17:55:36,625 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/90fe31c34a02480a89062be6de63b950 2023-03-22 17:55:36,630 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/3e749f660de744d1b6cc71cee50c76c2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/3e749f660de744d1b6cc71cee50c76c2 2023-03-22 17:55:36,637 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/3e749f660de744d1b6cc71cee50c76c2, entries=150, sequenceid=291, filesize=25.4 K 2023-03-22 17:55:36,638 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/76f081d212104588b46195af36a0525d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/76f081d212104588b46195af36a0525d 2023-03-22 17:55:36,643 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/76f081d212104588b46195af36a0525d, entries=150, sequenceid=291, filesize=11.9 K 2023-03-22 17:55:36,644 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/90fe31c34a02480a89062be6de63b950 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/90fe31c34a02480a89062be6de63b950 2023-03-22 17:55:36,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:36,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507796642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:36,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507796644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,651 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/90fe31c34a02480a89062be6de63b950, entries=150, sequenceid=291, filesize=11.9 K 2023-03-22 17:55:36,651 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 2343ed3ff2707ca3be5523fce8688282 in 1298ms, sequenceid=291, compaction requested=true 2023-03-22 17:55:36,651 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:36,652 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:36,653 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,653 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:36,654 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:36,654 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:36,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:36,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:36,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@629a7e01[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:36,655 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:36,656 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:36,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:36,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:36,661 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:36,661 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:36,661 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:36,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:36,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:36,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:36,663 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:36,666 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:36,666 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:36,666 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:36,666 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:36,666 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:36,668 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,668 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,669 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:36,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:36,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:36,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:36,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:36,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:36,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:36,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:36,673 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:36,673 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:36,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:36,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:36,673 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:36,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:36,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:36,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:36,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:36,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:36,773 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:36,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:36,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:36,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:36,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:36,785 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:36,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:36,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:36,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:36,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,786 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:36,793 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:36,793 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:36,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:36,793 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:36,793 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:36,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:36,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:36,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:36,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:36,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:36,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:36,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:36,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:36,795 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:36,795 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:36,795 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:36,795 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:36,795 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:55:36,795 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:36,796 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:36,796 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:36,796 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:36,796 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:36,796 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:36,817 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303223b966c63a75347119ce2a578642466af to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223b966c63a75347119ce2a578642466af 2023-03-22 17:55:37,232 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=302, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/2e6132befa0d41f1b30d7d8171da8abd 2023-03-22 17:55:37,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:37,665 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:37,666 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/fc6c880fb5a348b189bfe7be31af5954 2023-03-22 17:55:37,675 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/bab2470858da4ae8b9c3b1fc0c2dbe6b 2023-03-22 17:55:37,678 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/2e6132befa0d41f1b30d7d8171da8abd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2e6132befa0d41f1b30d7d8171da8abd 2023-03-22 17:55:37,682 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2e6132befa0d41f1b30d7d8171da8abd, entries=100, sequenceid=302, filesize=18.6 K 2023-03-22 17:55:37,683 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/fc6c880fb5a348b189bfe7be31af5954 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/fc6c880fb5a348b189bfe7be31af5954 2023-03-22 17:55:37,686 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/fc6c880fb5a348b189bfe7be31af5954, entries=100, sequenceid=302, filesize=9.5 K 2023-03-22 17:55:37,687 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/bab2470858da4ae8b9c3b1fc0c2dbe6b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bab2470858da4ae8b9c3b1fc0c2dbe6b 2023-03-22 17:55:37,690 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bab2470858da4ae8b9c3b1fc0c2dbe6b, entries=100, sequenceid=302, filesize=9.5 K 2023-03-22 17:55:37,691 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for 2343ed3ff2707ca3be5523fce8688282 in 895ms, sequenceid=302, compaction requested=true 2023-03-22 17:55:37,691 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:37,691 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:37,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,693 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,693 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:37,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2652ea2f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:37,694 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:37,694 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,695 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:37,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,695 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:37,697 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:37,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,697 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:37,699 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:37,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:37,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:37,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:37,700 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:37,707 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,707 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,707 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,707 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,708 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,710 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:37,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:37,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:37,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:37,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:37,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:37,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:37,711 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:37,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:37,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,711 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:37,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:37,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:37,716 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:37,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:37,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:37,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:37,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:37,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:37,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:37,728 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303221fa48b288ffb4a269d6b6abdaede148c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303221fa48b288ffb4a269d6b6abdaede148c 2023-03-22 17:55:37,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507737731, max timestamp=1679507737790Num uniques 859; 2023-03-22 17:55:37,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507737731, max timestamp=1679507737790Num uniques 859; 2023-03-22 17:55:37,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507737731, max timestamp=1679507737790Num uniques 859; 2023-03-22 17:55:37,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:37,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:37,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:37,811 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:37,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:37,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:37,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:37,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:37,815 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:37,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:37,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:37,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:37,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,816 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,816 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:37,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:37,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,817 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:37,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:37,818 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:37,818 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:37,818 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:37,818 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:37,818 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:37,820 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:37,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a50ff6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:37,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:37,822 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,824 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:37,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:37,824 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,824 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:37,824 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:37,825 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:37,825 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,825 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,825 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:37,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:37,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:37,828 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:37,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:37,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:37,828 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:37,828 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:37,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,833 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:37,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:37,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:37,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:37,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:37,836 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:37,836 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:37,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:37,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:37,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:37,836 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:37,836 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,836 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,836 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:37,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507797850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:37,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507797851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,936 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:37,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:37,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:37,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:37,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:37,939 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:37,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:37,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:37,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:37,940 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,940 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,942 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:37,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:37,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:37,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:37,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:37,943 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:37,943 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:37,943 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:37,944 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:37,944 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:37,944 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:37,944 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:37,945 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,946 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,946 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:37,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288cb717[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:37,947 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:37,947 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,948 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,948 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:37,948 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:37,954 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:37,954 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:37,954 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,954 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:37,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:37,957 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:37,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:37,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:37,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:37,958 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:37,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,959 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:37,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:37,959 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,961 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:37,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:37,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:37,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:37,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:37,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:37,963 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:37,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:37,963 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:37,963 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:37,964 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:37,964 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:37,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:37,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507797957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:37,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:37,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507797960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,064 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:38,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:38,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:38,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:38,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:38,068 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:38,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:38,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:38,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:38,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,069 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:38,070 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,070 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:38,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:38,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:38,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:38,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:38,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:38,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:38,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:38,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:38,072 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:38,072 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:38,072 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:38,072 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:38,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:38,072 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:38,073 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,073 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:38,073 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,075 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:38,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:38,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1957be71[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:38,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:38,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:38,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:38,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:38,079 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:38,080 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:38,080 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:38,081 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:38,085 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:38,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:38,088 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:38,088 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:38,088 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:38,088 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:38,088 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:38,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,089 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:38,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:38,090 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,091 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:38,091 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:38,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:38,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:38,094 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:38,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:38,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:38,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:38,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:38,095 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:38,095 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:38,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:38,134 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=312, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/9a4b4e04a4b1490b8896695926a3539d 2023-03-22 17:55:38,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:38,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507798165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:38,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507798166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/0ce4eec1959f430b8dd0d9e5c314acbb 2023-03-22 17:55:38,195 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:38,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:38,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:38,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:38,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:38,216 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:38,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:38,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:38,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:38,217 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,217 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:38,218 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:38,218 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,218 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:38,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:38,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:38,256 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:38,256 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:38,256 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:38,257 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:38,257 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:38,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/f64f8dbca2604ea2a9fe6e767b5c9b0d 2023-03-22 17:55:38,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/9a4b4e04a4b1490b8896695926a3539d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9a4b4e04a4b1490b8896695926a3539d 2023-03-22 17:55:38,264 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:38,264 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:38,264 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:38,264 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:38,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:38,264 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:38,265 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,265 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:38,265 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:38,266 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,266 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:38,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:38,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:38,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28652ce6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:38,269 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:38,269 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,269 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9a4b4e04a4b1490b8896695926a3539d, entries=200, sequenceid=312, filesize=32.3 K 2023-03-22 17:55:38,270 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,270 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:38,270 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:38,270 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:38,270 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/0ce4eec1959f430b8dd0d9e5c314acbb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0ce4eec1959f430b8dd0d9e5c314acbb 2023-03-22 17:55:38,270 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,270 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:38,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:38,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:38,272 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:38,275 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0ce4eec1959f430b8dd0d9e5c314acbb, entries=150, sequenceid=312, filesize=11.9 K 2023-03-22 17:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:38,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/f64f8dbca2604ea2a9fe6e767b5c9b0d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f64f8dbca2604ea2a9fe6e767b5c9b0d 2023-03-22 17:55:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:38,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:38,277 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:38,277 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:38,277 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:38,277 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:38,277 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:38,279 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,279 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:38,279 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,280 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,283 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:38,283 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:38,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:38,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:38,284 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,284 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,284 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,284 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:38,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:38,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:38,284 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:38,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,285 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:38,285 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:38,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:38,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f64f8dbca2604ea2a9fe6e767b5c9b0d, entries=150, sequenceid=312, filesize=11.9 K 2023-03-22 17:55:38,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2343ed3ff2707ca3be5523fce8688282 in 590ms, sequenceid=312, compaction requested=true 2023-03-22 17:55:38,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:38,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:38,306 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-22 17:55:38,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:38,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:38,306 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-22 17:55:38,309 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156194 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-22 17:55:38,309 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70598 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-22 17:55:38,309 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/A is initiating minor compaction (all files) 2023-03-22 17:55:38,310 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/B is initiating minor compaction (all files) 2023-03-22 17:55:38,310 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/A in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:38,310 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/B in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:38,310 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/227916012d0f4b14bed3eac56152b8bb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/036eacfb09c24bcd858b43b1e7e6bc55, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9586133263a24923b63312a3abb77aa2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/3e749f660de744d1b6cc71cee50c76c2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2e6132befa0d41f1b30d7d8171da8abd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9a4b4e04a4b1490b8896695926a3539d] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=152.5 K 2023-03-22 17:55:38,310 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/152a63f9c8a74511a28e9d1ce96285a8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d214720099d44b9bbe7a4168a980f872, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7cbc0ea7d73b4388a5305584bc2f3656, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/76f081d212104588b46195af36a0525d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/fc6c880fb5a348b189bfe7be31af5954, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0ce4eec1959f430b8dd0d9e5c314acbb] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=68.9 K 2023-03-22 17:55:38,310 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 227916012d0f4b14bed3eac56152b8bb, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507730839 2023-03-22 17:55:38,310 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 152a63f9c8a74511a28e9d1ce96285a8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507730839 2023-03-22 17:55:38,311 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 036eacfb09c24bcd858b43b1e7e6bc55, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679507730939 2023-03-22 17:55:38,311 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d214720099d44b9bbe7a4168a980f872, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679507730939 2023-03-22 17:55:38,311 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 9586133263a24923b63312a3abb77aa2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1679507733103 2023-03-22 17:55:38,311 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7cbc0ea7d73b4388a5305584bc2f3656, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1679507733103 2023-03-22 17:55:38,312 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 3e749f660de744d1b6cc71cee50c76c2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679507734345 2023-03-22 17:55:38,312 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2e6132befa0d41f1b30d7d8171da8abd, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679507735511 2023-03-22 17:55:38,312 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 76f081d212104588b46195af36a0525d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679507734345 2023-03-22 17:55:38,313 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 9a4b4e04a4b1490b8896695926a3539d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679507737681 2023-03-22 17:55:38,313 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting fc6c880fb5a348b189bfe7be31af5954, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679507735511 2023-03-22 17:55:38,313 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0ce4eec1959f430b8dd0d9e5c314acbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679507737706 2023-03-22 17:55:38,349 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#B#compaction#352 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:38,372 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#A#compaction#351 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:38,385 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:38,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:38,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:38,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:38,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:38,432 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:38,432 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:38,432 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:38,432 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:38,433 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,433 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,434 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:38,434 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:38,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:38,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:38,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:38,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:38,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:38,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:38,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:38,453 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:38,453 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:38,454 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:38,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:38,455 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-22 17:55:38,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:38,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:38,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:38,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:38,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:38,455 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:38,459 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/b36976dfa5644a5abe3ff420e974aded as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b36976dfa5644a5abe3ff420e974aded 2023-03-22 17:55:38,465 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/B of 2343ed3ff2707ca3be5523fce8688282 into b36976dfa5644a5abe3ff420e974aded(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:38,465 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:38,465 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/B, priority=10, startTime=1679507738306; duration=0sec 2023-03-22 17:55:38,465 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:38,465 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-22 17:55:38,467 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70598 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-22 17:55:38,467 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/C is initiating minor compaction (all files) 2023-03-22 17:55:38,467 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/C in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:38,468 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b9172304f834b42b076ff5ca2531649, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1ed2b2b35002448b95842852a12a4c02, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/41517e3c24b14536b1683a71426422ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/90fe31c34a02480a89062be6de63b950, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bab2470858da4ae8b9c3b1fc0c2dbe6b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f64f8dbca2604ea2a9fe6e767b5c9b0d] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=68.9 K 2023-03-22 17:55:38,468 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4b9172304f834b42b076ff5ca2531649, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507730839 2023-03-22 17:55:38,469 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1ed2b2b35002448b95842852a12a4c02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679507730939 2023-03-22 17:55:38,469 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 41517e3c24b14536b1683a71426422ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1679507733103 2023-03-22 17:55:38,469 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 90fe31c34a02480a89062be6de63b950, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679507734345 2023-03-22 17:55:38,470 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting bab2470858da4ae8b9c3b1fc0c2dbe6b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679507735511 2023-03-22 17:55:38,470 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f64f8dbca2604ea2a9fe6e767b5c9b0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679507737706 2023-03-22 17:55:38,488 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#C#compaction#354 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:38,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:38,494 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:38,513 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/47b32e7ee8da4f43b384c4d31a9f6402 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/47b32e7ee8da4f43b384c4d31a9f6402 2023-03-22 17:55:38,517 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/C of 2343ed3ff2707ca3be5523fce8688282 into 47b32e7ee8da4f43b384c4d31a9f6402(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:38,517 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:38,517 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/C, priority=10, startTime=1679507738306; duration=0sec 2023-03-22 17:55:38,517 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:38,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:38,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507798515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:38,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507798524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:38,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507798625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:38,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507798634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:38,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507798830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:38,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507798842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:38,910 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322961ca70dce9f459289e53d48940dd745 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322961ca70dce9f459289e53d48940dd745 2023-03-22 17:55:38,912 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/5672101319c44490bf9c407c04f131c5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5672101319c44490bf9c407c04f131c5 2023-03-22 17:55:38,916 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/A of 2343ed3ff2707ca3be5523fce8688282 into 5672101319c44490bf9c407c04f131c5(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:38,916 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:38,916 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/A, priority=10, startTime=1679507738306; duration=0sec 2023-03-22 17:55:38,917 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:39,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:39,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507799137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:39,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40272 deadline: 1679507799137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,142 DEBUG [Thread-2381] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:55:39,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:39,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507799147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,316 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/490b333659494e9ca56d95edea1e8715 2023-03-22 17:55:39,325 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/09db8b5ca9904d92a9fcff38b6edc9c8 2023-03-22 17:55:39,504 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-22 17:55:39,535 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/0b3b6df8495742ee85a266ad2e906962, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/abf29c70133f43e2a5cd42d5a5e2cd0d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5da3fd8bbb374a16a5b5b29ebd30c54a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e73ed84e4523404b97f43adf20b62be3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a49d3ec542d2489b86761a0adda1dd1d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b01a853384524d3f9bc86562737a3ecb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/274ad8987591402d8a0ae5aa8f340e90, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/82c75fb4ca1e406b992672eaacf6a3c6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/1a6836d282c94bd7861cd3f688fc16a6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/72ed888691da42e09320b353a36fc6af, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e287a64c6c5b456694c891409220ffb8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/227916012d0f4b14bed3eac56152b8bb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/036eacfb09c24bcd858b43b1e7e6bc55, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9586133263a24923b63312a3abb77aa2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/3e749f660de744d1b6cc71cee50c76c2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2e6132befa0d41f1b30d7d8171da8abd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9a4b4e04a4b1490b8896695926a3539d] to archive 2023-03-22 17:55:39,536 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:39,538 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/0b3b6df8495742ee85a266ad2e906962 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/0b3b6df8495742ee85a266ad2e906962 2023-03-22 17:55:39,539 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/abf29c70133f43e2a5cd42d5a5e2cd0d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/abf29c70133f43e2a5cd42d5a5e2cd0d 2023-03-22 17:55:39,540 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5da3fd8bbb374a16a5b5b29ebd30c54a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5da3fd8bbb374a16a5b5b29ebd30c54a 2023-03-22 17:55:39,542 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e73ed84e4523404b97f43adf20b62be3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e73ed84e4523404b97f43adf20b62be3 2023-03-22 17:55:39,543 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a49d3ec542d2489b86761a0adda1dd1d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a49d3ec542d2489b86761a0adda1dd1d 2023-03-22 17:55:39,544 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b01a853384524d3f9bc86562737a3ecb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b01a853384524d3f9bc86562737a3ecb 2023-03-22 17:55:39,545 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/274ad8987591402d8a0ae5aa8f340e90 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/274ad8987591402d8a0ae5aa8f340e90 2023-03-22 17:55:39,546 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/82c75fb4ca1e406b992672eaacf6a3c6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/82c75fb4ca1e406b992672eaacf6a3c6 2023-03-22 17:55:39,547 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/1a6836d282c94bd7861cd3f688fc16a6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/1a6836d282c94bd7861cd3f688fc16a6 2023-03-22 17:55:39,548 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/72ed888691da42e09320b353a36fc6af to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/72ed888691da42e09320b353a36fc6af 2023-03-22 17:55:39,549 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e287a64c6c5b456694c891409220ffb8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/e287a64c6c5b456694c891409220ffb8 2023-03-22 17:55:39,550 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/227916012d0f4b14bed3eac56152b8bb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/227916012d0f4b14bed3eac56152b8bb 2023-03-22 17:55:39,551 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/036eacfb09c24bcd858b43b1e7e6bc55 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/036eacfb09c24bcd858b43b1e7e6bc55 2023-03-22 17:55:39,552 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9586133263a24923b63312a3abb77aa2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9586133263a24923b63312a3abb77aa2 2023-03-22 17:55:39,554 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/3e749f660de744d1b6cc71cee50c76c2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/3e749f660de744d1b6cc71cee50c76c2 2023-03-22 17:55:39,555 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2e6132befa0d41f1b30d7d8171da8abd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2e6132befa0d41f1b30d7d8171da8abd 2023-03-22 17:55:39,556 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9a4b4e04a4b1490b8896695926a3539d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/9a4b4e04a4b1490b8896695926a3539d 2023-03-22 17:55:39,570 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/043afd621392438da094309a544f24c0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/91f7f1776bfd4b12ae2b2b8ac088b782, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0a417a4d20064d22908857c545fc8cb1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/083b5d3c4f1a4c1399f8258f743c76da, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/4492214b7dc447f6a9ee3a708ab85d37, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/8fe4ff785f8d4a31b02a699f21a8809c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/03ee4005700a40acaf7f209b1d9dee4f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/1c9f05de8aef42ce9a555b3623714df4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b3b6cd328aa34c099c1abf7087439bcd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/5695858e752f4874acca3612f01d7f1f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/152a63f9c8a74511a28e9d1ce96285a8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/24e9cc124d924cf8aac4aeeddb423280, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d214720099d44b9bbe7a4168a980f872, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7cbc0ea7d73b4388a5305584bc2f3656, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/76f081d212104588b46195af36a0525d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/fc6c880fb5a348b189bfe7be31af5954, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0ce4eec1959f430b8dd0d9e5c314acbb] to archive 2023-03-22 17:55:39,571 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:39,572 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/043afd621392438da094309a544f24c0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/043afd621392438da094309a544f24c0 2023-03-22 17:55:39,573 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/91f7f1776bfd4b12ae2b2b8ac088b782 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/91f7f1776bfd4b12ae2b2b8ac088b782 2023-03-22 17:55:39,574 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0a417a4d20064d22908857c545fc8cb1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0a417a4d20064d22908857c545fc8cb1 2023-03-22 17:55:39,576 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/083b5d3c4f1a4c1399f8258f743c76da to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/083b5d3c4f1a4c1399f8258f743c76da 2023-03-22 17:55:39,577 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/4492214b7dc447f6a9ee3a708ab85d37 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/4492214b7dc447f6a9ee3a708ab85d37 2023-03-22 17:55:39,578 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/8fe4ff785f8d4a31b02a699f21a8809c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/8fe4ff785f8d4a31b02a699f21a8809c 2023-03-22 17:55:39,579 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/03ee4005700a40acaf7f209b1d9dee4f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/03ee4005700a40acaf7f209b1d9dee4f 2023-03-22 17:55:39,580 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/1c9f05de8aef42ce9a555b3623714df4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/1c9f05de8aef42ce9a555b3623714df4 2023-03-22 17:55:39,581 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b3b6cd328aa34c099c1abf7087439bcd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b3b6cd328aa34c099c1abf7087439bcd 2023-03-22 17:55:39,582 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/5695858e752f4874acca3612f01d7f1f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/5695858e752f4874acca3612f01d7f1f 2023-03-22 17:55:39,584 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/152a63f9c8a74511a28e9d1ce96285a8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/152a63f9c8a74511a28e9d1ce96285a8 2023-03-22 17:55:39,585 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/24e9cc124d924cf8aac4aeeddb423280 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/24e9cc124d924cf8aac4aeeddb423280 2023-03-22 17:55:39,586 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d214720099d44b9bbe7a4168a980f872 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d214720099d44b9bbe7a4168a980f872 2023-03-22 17:55:39,587 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7cbc0ea7d73b4388a5305584bc2f3656 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7cbc0ea7d73b4388a5305584bc2f3656 2023-03-22 17:55:39,589 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/76f081d212104588b46195af36a0525d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/76f081d212104588b46195af36a0525d 2023-03-22 17:55:39,591 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/fc6c880fb5a348b189bfe7be31af5954 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/fc6c880fb5a348b189bfe7be31af5954 2023-03-22 17:55:39,594 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0ce4eec1959f430b8dd0d9e5c314acbb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/0ce4eec1959f430b8dd0d9e5c314acbb 2023-03-22 17:55:39,599 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d3d0ee4992d846e9bd926a689697a5b3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/6f6d6b193e894d31b229d7a6d615dfc0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/54a086e85a1740ab88efd79daa00293a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f3b915c14f3e4f0993f6979323704ef7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1b6178afd83845a092922cf21da7442d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/125705772b5f439980dc6fdc92a3fa1a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/93c5db0af66948b09241a3ee20705f4e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/8a951bef60294f73b1ade66038ec8671, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/a234ad8f33674d92b1032d8561a7bebd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b8f6d6f79684e639353462747f95a42, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b9172304f834b42b076ff5ca2531649, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4ac7b030e5ee46d3aed4c78e2c5ac5f7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1ed2b2b35002448b95842852a12a4c02, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/41517e3c24b14536b1683a71426422ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/90fe31c34a02480a89062be6de63b950, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bab2470858da4ae8b9c3b1fc0c2dbe6b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f64f8dbca2604ea2a9fe6e767b5c9b0d] to archive 2023-03-22 17:55:39,600 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:39,601 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d3d0ee4992d846e9bd926a689697a5b3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d3d0ee4992d846e9bd926a689697a5b3 2023-03-22 17:55:39,603 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/6f6d6b193e894d31b229d7a6d615dfc0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/6f6d6b193e894d31b229d7a6d615dfc0 2023-03-22 17:55:39,604 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/54a086e85a1740ab88efd79daa00293a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/54a086e85a1740ab88efd79daa00293a 2023-03-22 17:55:39,607 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f3b915c14f3e4f0993f6979323704ef7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f3b915c14f3e4f0993f6979323704ef7 2023-03-22 17:55:39,608 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1b6178afd83845a092922cf21da7442d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1b6178afd83845a092922cf21da7442d 2023-03-22 17:55:39,609 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/125705772b5f439980dc6fdc92a3fa1a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/125705772b5f439980dc6fdc92a3fa1a 2023-03-22 17:55:39,610 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/93c5db0af66948b09241a3ee20705f4e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/93c5db0af66948b09241a3ee20705f4e 2023-03-22 17:55:39,611 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/8a951bef60294f73b1ade66038ec8671 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/8a951bef60294f73b1ade66038ec8671 2023-03-22 17:55:39,612 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/a234ad8f33674d92b1032d8561a7bebd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/a234ad8f33674d92b1032d8561a7bebd 2023-03-22 17:55:39,614 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b8f6d6f79684e639353462747f95a42 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b8f6d6f79684e639353462747f95a42 2023-03-22 17:55:39,615 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b9172304f834b42b076ff5ca2531649 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4b9172304f834b42b076ff5ca2531649 2023-03-22 17:55:39,616 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4ac7b030e5ee46d3aed4c78e2c5ac5f7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4ac7b030e5ee46d3aed4c78e2c5ac5f7 2023-03-22 17:55:39,617 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1ed2b2b35002448b95842852a12a4c02 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/1ed2b2b35002448b95842852a12a4c02 2023-03-22 17:55:39,618 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/41517e3c24b14536b1683a71426422ae to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/41517e3c24b14536b1683a71426422ae 2023-03-22 17:55:39,619 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/90fe31c34a02480a89062be6de63b950 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/90fe31c34a02480a89062be6de63b950 2023-03-22 17:55:39,620 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bab2470858da4ae8b9c3b1fc0c2dbe6b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bab2470858da4ae8b9c3b1fc0c2dbe6b 2023-03-22 17:55:39,621 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f64f8dbca2604ea2a9fe6e767b5c9b0d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/f64f8dbca2604ea2a9fe6e767b5c9b0d 2023-03-22 17:55:39,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:39,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507799645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:39,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507799652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,735 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/0ce43f8443dc4cafb1724b355d2d65fe 2023-03-22 17:55:39,740 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/490b333659494e9ca56d95edea1e8715 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/490b333659494e9ca56d95edea1e8715 2023-03-22 17:55:39,745 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/490b333659494e9ca56d95edea1e8715, entries=150, sequenceid=348, filesize=25.4 K 2023-03-22 17:55:39,746 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/09db8b5ca9904d92a9fcff38b6edc9c8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/09db8b5ca9904d92a9fcff38b6edc9c8 2023-03-22 17:55:39,749 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/09db8b5ca9904d92a9fcff38b6edc9c8, entries=150, sequenceid=348, filesize=11.9 K 2023-03-22 17:55:39,750 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/0ce43f8443dc4cafb1724b355d2d65fe as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/0ce43f8443dc4cafb1724b355d2d65fe 2023-03-22 17:55:39,753 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/0ce43f8443dc4cafb1724b355d2d65fe, entries=150, sequenceid=348, filesize=11.9 K 2023-03-22 17:55:39,754 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2343ed3ff2707ca3be5523fce8688282 in 1299ms, sequenceid=348, compaction requested=false 2023-03-22 17:55:39,754 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:39,754 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:39,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:39,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:39,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:39,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:39,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:39,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:39,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7104ca3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:39,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:39,760 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:39,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:39,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:39,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:39,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:39,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:39,761 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,761 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:39,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:39,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:39,762 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,762 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:39,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:39,765 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:39,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:39,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:39,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:39,765 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:39,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:39,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,767 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:39,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:39,770 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:39,771 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:39,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:39,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:39,777 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:39,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:39,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:39,777 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:39,777 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:39,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:39,878 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:39,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:39,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:39,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:39,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:39,881 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:39,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:39,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:39,882 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:39,884 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,884 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:39,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:39,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:39,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:39,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:39,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:39,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:39,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:39,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:39,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:39,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:39,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:39,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:39,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:39,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:39,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:39,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:39,888 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:39,888 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:39,888 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:39,888 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:39,888 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:39,888 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:39,889 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:39,889 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:39,889 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:39,889 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:39,904 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322464fe4fbb5064c2a9e2ab1815a714fb7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322464fe4fbb5064c2a9e2ab1815a714fb7 2023-03-22 17:55:39,909 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=361, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/8faa83002fa04797872fe0ede6bc4db3 2023-03-22 17:55:39,919 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/7b5d9206d2bf4c3c9e788a6d7697aff9 2023-03-22 17:55:40,328 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/56c3213436a94acda0a4c2bb039092ff 2023-03-22 17:55:40,332 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/8faa83002fa04797872fe0ede6bc4db3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/8faa83002fa04797872fe0ede6bc4db3 2023-03-22 17:55:40,336 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/8faa83002fa04797872fe0ede6bc4db3, entries=150, sequenceid=361, filesize=25.4 K 2023-03-22 17:55:40,336 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/7b5d9206d2bf4c3c9e788a6d7697aff9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7b5d9206d2bf4c3c9e788a6d7697aff9 2023-03-22 17:55:40,340 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7b5d9206d2bf4c3c9e788a6d7697aff9, entries=150, sequenceid=361, filesize=11.9 K 2023-03-22 17:55:40,340 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/56c3213436a94acda0a4c2bb039092ff as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/56c3213436a94acda0a4c2bb039092ff 2023-03-22 17:55:40,344 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/56c3213436a94acda0a4c2bb039092ff, entries=150, sequenceid=361, filesize=11.9 K 2023-03-22 17:55:40,344 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 2343ed3ff2707ca3be5523fce8688282 in 456ms, sequenceid=361, compaction requested=true 2023-03-22 17:55:40,344 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:40,344 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:40,344 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:40,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:40,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:40,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:40,346 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:40,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae96759[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:40,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:40,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,349 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:40,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:40,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:40,349 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:40,352 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:40,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:40,354 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:40,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:40,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:40,355 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:40,355 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:40,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:40,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:40,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:40,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,360 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:40,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,362 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:40,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:40,362 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:40,363 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:40,363 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:40,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:40,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:40,463 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:40,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:40,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:40,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:40,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:40,466 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:40,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:40,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:40,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:40,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,468 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:40,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:40,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,468 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:40,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:40,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:40,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:40,469 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:40,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:40,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:40,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:40,470 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:40,470 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:40,472 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,472 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:40,474 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:40,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee9fc43[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:40,474 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,475 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:40,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,476 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:40,477 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:40,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:40,479 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:40,480 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:40,480 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:40,480 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:40,480 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:40,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:40,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,483 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,486 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,486 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,486 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:40,487 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:40,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:40,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:40,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:40,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:40,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:40,487 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:40,487 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:40,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:40,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:40,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:40,587 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:40,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:40,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:40,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:40,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:40,590 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:40,590 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:40,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:40,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:40,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,591 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,592 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:40,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:40,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:40,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:40,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:40,594 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:40,594 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:40,594 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:40,594 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:40,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:40,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,596 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,596 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6feaff64[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:40,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:40,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,599 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:40,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:40,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,599 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:40,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:40,602 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,602 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:40,602 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,602 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:40,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:40,604 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:40,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:40,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:40,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:40,604 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:40,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:40,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:40,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:40,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:40,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:40,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:40,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,608 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,613 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,613 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,613 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:40,613 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:40,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:40,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:40,613 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:40,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:40,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:40,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:40,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,614 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:40,614 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:40,714 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:40,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:40,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:40,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:40,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:40,718 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:40,718 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:40,718 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:40,718 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:40,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:40,721 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:40,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:40,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:40,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:40,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:40,722 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:40,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:40,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:40,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:40,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:40,724 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-22 17:55:40,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:40,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:40,725 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:40,725 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:40,725 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:40,725 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:40,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:40,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:40,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507740775, max timestamp=1679507740794Num uniques 859; 2023-03-22 17:55:40,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:40,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507740775, max timestamp=1679507740794Num uniques 859; 2023-03-22 17:55:40,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507740775, max timestamp=1679507740794Num uniques 859; 2023-03-22 17:55:40,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:40,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:40,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507740830, max timestamp=1679507740845Num uniques 1068; 2023-03-22 17:55:40,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-22 17:55:40,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507740830, max timestamp=1679507740845Num uniques 1068; 2023-03-22 17:55:40,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-22 17:55:40,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679507740830, max timestamp=1679507740845Num uniques 1068; 2023-03-22 17:55:40,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-22 17:55:40,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507740854, max timestamp=1679507740897Num uniques 1027; 2023-03-22 17:55:40,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-22 17:55:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507740854, max timestamp=1679507740897Num uniques 1027; 2023-03-22 17:55:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-22 17:55:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679507740854, max timestamp=1679507740897Num uniques 1027; 2023-03-22 17:55:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-22 17:55:40,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507740938, max timestamp=1679507740959Num uniques 1036; 2023-03-22 17:55:40,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-22 17:55:40,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507740938, max timestamp=1679507740959Num uniques 1036; 2023-03-22 17:55:40,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-22 17:55:40,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679507740938, max timestamp=1679507740959Num uniques 1036; 2023-03-22 17:55:40,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-22 17:55:41,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507741001, max timestamp=1679507741019Num uniques 1095; 2023-03-22 17:55:41,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507741001, max timestamp=1679507741019Num uniques 1095; 2023-03-22 17:55:41,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-22 17:55:41,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-22 17:55:41,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679507741001, max timestamp=1679507741019Num uniques 1095; 2023-03-22 17:55:41,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-22 17:55:41,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679507741057, max timestamp=1679507741080Num uniques 1104; 2023-03-22 17:55:41,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679507741057, max timestamp=1679507741080Num uniques 1104; 2023-03-22 17:55:41,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-22 17:55:41,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-22 17:55:41,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679507741057, max timestamp=1679507741080Num uniques 1104; 2023-03-22 17:55:41,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-22 17:55:41,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679507741099, max timestamp=1679507741134Num uniques 1113; 2023-03-22 17:55:41,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-22 17:55:41,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679507741099, max timestamp=1679507741134Num uniques 1113; 2023-03-22 17:55:41,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679507741099, max timestamp=1679507741134Num uniques 1113; 2023-03-22 17:55:41,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-22 17:55:41,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-22 17:55:41,140 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303221f28bc78bc944a94bb7d664b3cf4f000 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303221f28bc78bc944a94bb7d664b3cf4f000 2023-03-22 17:55:41,145 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=369, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/2eab2e03eafd41ad8898589d3a5c4319 2023-03-22 17:55:41,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679507741175, max timestamp=1679507741223Num uniques 1122; 2023-03-22 17:55:41,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679507741175, max timestamp=1679507741223Num uniques 1122; 2023-03-22 17:55:41,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679507741175, max timestamp=1679507741223Num uniques 1122; 2023-03-22 17:55:41,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-03-22 17:55:41,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-03-22 17:55:41,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-03-22 17:55:41,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8276, totalHeapSize=19064, min timestamp=1679507741290, max timestamp=1679507741306Num uniques 1081; 2023-03-22 17:55:41,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49496, new segment data size=8276, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-22 17:55:41,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8276, totalHeapSize=19064, min timestamp=1679507741290, max timestamp=1679507741306Num uniques 1081; 2023-03-22 17:55:41,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49496, new segment data size=8276, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-22 17:55:41,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8279, totalHeapSize=19064, min timestamp=1679507741290, max timestamp=1679507741306Num uniques 1081; 2023-03-22 17:55:41,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49499, new segment data size=8279, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-22 17:55:41,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679507741342, max timestamp=1679507741365Num uniques 1040; 2023-03-22 17:55:41,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679507741342, max timestamp=1679507741365Num uniques 1040; 2023-03-22 17:55:41,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-22 17:55:41,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-22 17:55:41,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679507741342, max timestamp=1679507741365Num uniques 1040; 2023-03-22 17:55:41,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-22 17:55:41,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679507741406, max timestamp=1679507741439Num uniques 1049; 2023-03-22 17:55:41,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-22 17:55:41,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679507741406, max timestamp=1679507741439Num uniques 1049; 2023-03-22 17:55:41,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-22 17:55:41,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679507741406, max timestamp=1679507741439Num uniques 1049; 2023-03-22 17:55:41,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-22 17:55:41,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679507741485, max timestamp=1679507741511Num uniques 1058; 2023-03-22 17:55:41,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679507741485, max timestamp=1679507741511Num uniques 1058; 2023-03-22 17:55:41,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679507741485, max timestamp=1679507741511Num uniques 1058; 2023-03-22 17:55:41,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-22 17:55:41,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-22 17:55:41,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-22 17:55:41,567 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/e0ee8b296f704016954a995b20f84f33 2023-03-22 17:55:41,575 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/d7540f3c8b804c50b4fe3389261c84e1 2023-03-22 17:55:41,578 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/2eab2e03eafd41ad8898589d3a5c4319 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2eab2e03eafd41ad8898589d3a5c4319 2023-03-22 17:55:41,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679507741543, max timestamp=1679507741578Num uniques 1017; 2023-03-22 17:55:41,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679507741543, max timestamp=1679507741578Num uniques 1017; 2023-03-22 17:55:41,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-22 17:55:41,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-22 17:55:41,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679507741543, max timestamp=1679507741578Num uniques 1017; 2023-03-22 17:55:41,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-22 17:55:41,588 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2eab2e03eafd41ad8898589d3a5c4319, entries=150, sequenceid=369, filesize=25.4 K 2023-03-22 17:55:41,589 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/e0ee8b296f704016954a995b20f84f33 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/e0ee8b296f704016954a995b20f84f33 2023-03-22 17:55:41,592 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/e0ee8b296f704016954a995b20f84f33, entries=150, sequenceid=369, filesize=11.9 K 2023-03-22 17:55:41,593 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/d7540f3c8b804c50b4fe3389261c84e1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d7540f3c8b804c50b4fe3389261c84e1 2023-03-22 17:55:41,596 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d7540f3c8b804c50b4fe3389261c84e1, entries=150, sequenceid=369, filesize=11.9 K 2023-03-22 17:55:41,596 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=46.96 KB/48090 for 2343ed3ff2707ca3be5523fce8688282 in 872ms, sequenceid=369, compaction requested=true 2023-03-22 17:55:41,596 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:41,596 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:41,598 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,600 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=53.67 KB heapSize=145.17 KB 2023-03-22 17:55:41,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:41,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9ac36e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:41,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:41,600 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:41,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:41,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:41,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:41,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:41,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:41,601 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:41,603 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:41,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:41,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:41,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:41,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:41,604 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:41,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:41,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:41,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:41,604 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:41,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:41,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,608 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,614 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:41,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:41,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:41,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:41,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:41,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:41,617 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:41,617 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:41,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:41,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507741670, max timestamp=1679507741709Num uniques 859; 2023-03-22 17:55:41,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:41,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507741670, max timestamp=1679507741709Num uniques 859; 2023-03-22 17:55:41,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507741670, max timestamp=1679507741709Num uniques 859; 2023-03-22 17:55:41,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:41,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:41,717 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:41,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:41,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:41,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:41,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:41,720 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:41,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:41,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:41,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:41,721 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,721 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,722 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:41,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:41,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:41,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:41,723 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:41,723 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:41,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:41,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:41,725 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,725 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,725 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,726 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:41,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b48a2cb[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:41,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,729 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:41,729 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:41,730 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:41,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:41,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:41,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:41,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:41,733 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:41,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:41,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:41,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:41,733 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:41,735 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,735 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,735 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:41,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:41,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:41,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:41,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:41,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:41,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:41,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:41,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,740 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:41,740 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:41,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:41,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507801786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:41,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507801787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,840 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:41,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:41,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:41,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:41,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:41,843 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:41,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:41,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:41,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:41,844 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,844 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,846 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:41,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:41,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:41,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:41,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:41,847 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:41,847 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:41,847 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:41,847 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:41,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:41,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:41,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:41,848 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:41,849 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,849 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,849 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,850 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:41,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae25005[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:41,851 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:41,851 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,852 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,852 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:41,852 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:41,852 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:41,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:41,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,853 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:41,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:41,856 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:41,856 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:41,856 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:41,856 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:41,856 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:41,858 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,858 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,859 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:41,859 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:41,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,862 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:41,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:41,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:41,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:41,862 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:41,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:41,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507801896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:41,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507801896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,963 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:41,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:41,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:41,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:41,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:41,966 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:41,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:41,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:41,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:41,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,971 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:41,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:41,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,971 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:41,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:41,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:41,972 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:41,973 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:41,973 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:41,973 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:41,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,975 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:41,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:41,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@724b97e2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:41,977 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,977 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:41,977 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:41,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:41,978 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:41,980 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,980 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,980 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:41,980 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:41,982 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:41,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:41,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:41,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:41,982 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:41,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:41,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:41,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:41,984 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:41,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:41,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:41,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:41,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:41,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:41,988 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,988 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:41,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:41,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,017 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322107846fc1e6e4e708a6e37a8f0db8a0c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322107846fc1e6e4e708a6e37a8f0db8a0c 2023-03-22 17:55:42,022 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=601, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/b182e24c3f794e9a8bc76822449d7353 2023-03-22 17:55:42,089 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:42,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:42,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:42,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:42,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:42,093 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:42,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:42,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:42,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,094 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,095 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:42,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:42,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:42,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:42,097 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,097 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:42,097 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:42,097 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:42,099 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:42,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ef6bb76[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:42,101 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:42,102 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,103 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:42,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:42,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,105 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,105 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,105 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:42,109 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:42,109 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:42,109 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:42,109 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:42,109 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:42,111 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,111 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,111 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:42,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507802104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:42,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507802104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:42,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,119 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,119 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:42,119 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:42,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,119 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,219 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:42,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:42,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:42,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:42,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:42,223 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:42,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:42,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:42,224 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,224 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,225 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:42,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:42,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:42,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:42,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:42,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:42,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:42,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:42,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:42,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:42,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:42,228 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:42,230 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,230 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,230 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,230 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:42,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60060699[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:42,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:42,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,233 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:42,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:42,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:42,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:42,235 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:42,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:42,240 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:42,240 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:42,240 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:42,240 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:42,240 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:42,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,247 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:42,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:42,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,248 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:42,248 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:42,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,348 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:42,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:42,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:42,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:42,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:42,352 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:42,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:42,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:42,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,354 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:42,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:42,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:42,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:42,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:42,355 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:42,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:42,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:42,356 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:42,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:42,358 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,358 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@383c10e4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:42,360 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:42,360 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,366 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,367 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:42,367 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:42,367 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,367 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:42,367 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,368 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:42,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:42,373 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:42,373 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:42,373 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:42,373 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:42,373 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:42,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,376 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,377 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,380 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,380 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,380 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,380 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:42,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:42,380 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,381 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,381 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,381 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,381 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,381 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:42,381 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:42,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:42,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,380 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:42,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40226 deadline: 1679507802414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:42,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40284 deadline: 1679507802416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,430 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/131642d093404c2c91d7212bfa2f9ad7 2023-03-22 17:55:42,481 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:42,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:42,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:42,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:42,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:42,484 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:42,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:42,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:42,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,485 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,486 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,486 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,486 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,486 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:42,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:42,488 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,488 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:42,488 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:42,488 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:42,489 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,489 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,490 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16666486[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:42,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:42,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:42,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:42,492 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:42,495 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:42,498 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:42,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:42,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:42,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:42,498 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:42,500 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,500 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,500 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,500 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,503 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,503 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,503 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,503 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:42,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:42,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:42,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,503 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,503 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,504 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:42,504 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:42,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,604 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:42,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:42,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:42,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:42,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:42,607 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:42,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:42,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:42,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,609 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,609 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,609 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,609 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:42,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:42,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:42,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:42,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:42,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:42,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:42,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:42,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:42,610 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,610 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. started... 2023-03-22 17:55:42,611 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. as already flushing 2023-03-22 17:55:42,611 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:42,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:42,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:42,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:42,615 DEBUG [Thread-2392] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:55285 2023-03-22 17:55:42,615 DEBUG [Thread-2394] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:55285 2023-03-22 17:55:42,615 DEBUG [Thread-2392] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:42,615 DEBUG [Thread-2394] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:42,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:42,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d305037[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:42,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:42,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,620 DEBUG [Thread-2396] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:55285 2023-03-22 17:55:42,620 DEBUG [Thread-2396] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:42,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:42,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:42,623 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:42,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,623 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,623 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:42,626 DEBUG [Thread-2398] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:55285 2023-03-22 17:55:42,626 DEBUG [Thread-2398] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:42,626 DEBUG [Thread-2390] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:55285 2023-03-22 17:55:42,626 DEBUG [Thread-2390] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:42,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:42,725 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:42,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:42,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:42,725 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:42,725 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:42,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,727 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:42,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:42,729 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:42,731 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:42,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:42,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:42,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:42,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:42,731 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:42,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:42,731 DEBUG [Thread-2389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:42,731 DEBUG [Thread-2389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:42,832 DEBUG [Thread-2389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:42,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:42,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/d9c3a8c18e804e17ae83f1d7d9f2ad18 2023-03-22 17:55:42,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/b182e24c3f794e9a8bc76822449d7353 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b182e24c3f794e9a8bc76822449d7353 2023-03-22 17:55:42,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b182e24c3f794e9a8bc76822449d7353, entries=200, sequenceid=601, filesize=32.3 K 2023-03-22 17:55:42,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/131642d093404c2c91d7212bfa2f9ad7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/131642d093404c2c91d7212bfa2f9ad7 2023-03-22 17:55:42,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/131642d093404c2c91d7212bfa2f9ad7, entries=150, sequenceid=601, filesize=11.9 K 2023-03-22 17:55:42,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/d9c3a8c18e804e17ae83f1d7d9f2ad18 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d9c3a8c18e804e17ae83f1d7d9f2ad18 2023-03-22 17:55:42,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d9c3a8c18e804e17ae83f1d7d9f2ad18, entries=150, sequenceid=601, filesize=11.9 K 2023-03-22 17:55:42,864 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.05 KB/169008, currentSize=127.47 KB/130530 for 2343ed3ff2707ca3be5523fce8688282 in 1264ms, sequenceid=601, compaction requested=true 2023-03-22 17:55:42,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:42,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:42,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:42,865 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:55:42,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:42,865 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:55:42,866 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:55:42,866 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:55:42,866 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/A is initiating minor compaction (all files) 2023-03-22 17:55:42,866 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/B is initiating minor compaction (all files) 2023-03-22 17:55:42,866 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/A in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,867 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/B in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,867 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5672101319c44490bf9c407c04f131c5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/490b333659494e9ca56d95edea1e8715, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/8faa83002fa04797872fe0ede6bc4db3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2eab2e03eafd41ad8898589d3a5c4319, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b182e24c3f794e9a8bc76822449d7353] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=134.6 K 2023-03-22 17:55:42,867 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b36976dfa5644a5abe3ff420e974aded, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/09db8b5ca9904d92a9fcff38b6edc9c8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7b5d9206d2bf4c3c9e788a6d7697aff9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/e0ee8b296f704016954a995b20f84f33, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/131642d093404c2c91d7212bfa2f9ad7] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=60.1 K 2023-03-22 17:55:42,867 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5672101319c44490bf9c407c04f131c5, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679507737706 2023-03-22 17:55:42,867 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b36976dfa5644a5abe3ff420e974aded, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679507737706 2023-03-22 17:55:42,867 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 490b333659494e9ca56d95edea1e8715, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679507737830 2023-03-22 17:55:42,867 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 09db8b5ca9904d92a9fcff38b6edc9c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679507737830 2023-03-22 17:55:42,867 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 8faa83002fa04797872fe0ede6bc4db3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1679507738504 2023-03-22 17:55:42,868 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7b5d9206d2bf4c3c9e788a6d7697aff9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1679507738504 2023-03-22 17:55:42,868 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2eab2e03eafd41ad8898589d3a5c4319, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679507740679 2023-03-22 17:55:42,868 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e0ee8b296f704016954a995b20f84f33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679507740679 2023-03-22 17:55:42,868 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b182e24c3f794e9a8bc76822449d7353, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679507741578 2023-03-22 17:55:42,868 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 131642d093404c2c91d7212bfa2f9ad7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679507741590 2023-03-22 17:55:42,876 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#B#compaction#367 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:42,878 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#A#compaction#366 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:42,889 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/2d9acd43cb3e4e97b508d4376ad8a23f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/2d9acd43cb3e4e97b508d4376ad8a23f 2023-03-22 17:55:42,894 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/B of 2343ed3ff2707ca3be5523fce8688282 into 2d9acd43cb3e4e97b508d4376ad8a23f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:42,894 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:42,894 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/B, priority=11, startTime=1679507742865; duration=0sec 2023-03-22 17:55:42,894 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:42,894 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:55:42,896 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:55:42,896 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 2343ed3ff2707ca3be5523fce8688282/C is initiating minor compaction (all files) 2023-03-22 17:55:42,896 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2343ed3ff2707ca3be5523fce8688282/C in TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:42,896 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/47b32e7ee8da4f43b384c4d31a9f6402, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/0ce43f8443dc4cafb1724b355d2d65fe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/56c3213436a94acda0a4c2bb039092ff, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d7540f3c8b804c50b4fe3389261c84e1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d9c3a8c18e804e17ae83f1d7d9f2ad18] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp, totalSize=60.1 K 2023-03-22 17:55:42,896 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 47b32e7ee8da4f43b384c4d31a9f6402, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1679507737706 2023-03-22 17:55:42,896 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0ce43f8443dc4cafb1724b355d2d65fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679507737830 2023-03-22 17:55:42,897 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 56c3213436a94acda0a4c2bb039092ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1679507738504 2023-03-22 17:55:42,897 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d7540f3c8b804c50b4fe3389261c84e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679507740679 2023-03-22 17:55:42,897 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d9c3a8c18e804e17ae83f1d7d9f2ad18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679507741590 2023-03-22 17:55:42,903 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 2343ed3ff2707ca3be5523fce8688282#C#compaction#368 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:42,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:42,931 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-22 17:55:42,931 DEBUG [Thread-2379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6434ce29 to 127.0.0.1:55285 2023-03-22 17:55:42,931 DEBUG [Thread-2379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:42,932 DEBUG [Thread-2387] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:55285 2023-03-22 17:55:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:55:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:55:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:42,932 DEBUG [Thread-2387] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:42,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:55:43,298 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/a9cc3668fad34b1690e1436394a077a5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a9cc3668fad34b1690e1436394a077a5 2023-03-22 17:55:43,300 DEBUG [Thread-2385] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:55285 2023-03-22 17:55:43,300 DEBUG [Thread-2385] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:43,301 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/A of 2343ed3ff2707ca3be5523fce8688282 into a9cc3668fad34b1690e1436394a077a5(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:43,302 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:43,302 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/A, priority=11, startTime=1679507742865; duration=0sec 2023-03-22 17:55:43,302 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:43,310 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/bf5447d8a0bd43cb8ef4e5f50b2af746 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bf5447d8a0bd43cb8ef4e5f50b2af746 2023-03-22 17:55:43,313 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2343ed3ff2707ca3be5523fce8688282/C of 2343ed3ff2707ca3be5523fce8688282 into bf5447d8a0bd43cb8ef4e5f50b2af746(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:43,313 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:43,313 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282., storeName=2343ed3ff2707ca3be5523fce8688282/C, priority=11, startTime=1679507742865; duration=0sec 2023-03-22 17:55:43,313 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:43,344 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322a715049a82e743e4bc381915a5d900bc to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a715049a82e743e4bc381915a5d900bc 2023-03-22 17:55:43,748 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=640, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/4e2c7ab4b150463497b313125a92a161 2023-03-22 17:55:44,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/d0a542196c114e9982ba15f0942a4ce2 2023-03-22 17:55:44,566 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/4afd02401262421b830e75e2c538e7cb 2023-03-22 17:55:44,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/4e2c7ab4b150463497b313125a92a161 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/4e2c7ab4b150463497b313125a92a161 2023-03-22 17:55:44,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/4e2c7ab4b150463497b313125a92a161, entries=150, sequenceid=640, filesize=25.4 K 2023-03-22 17:55:44,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/d0a542196c114e9982ba15f0942a4ce2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d0a542196c114e9982ba15f0942a4ce2 2023-03-22 17:55:44,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d0a542196c114e9982ba15f0942a4ce2, entries=150, sequenceid=640, filesize=11.9 K 2023-03-22 17:55:44,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/4afd02401262421b830e75e2c538e7cb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4afd02401262421b830e75e2c538e7cb 2023-03-22 17:55:44,589 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4afd02401262421b830e75e2c538e7cb, entries=150, sequenceid=640, filesize=11.9 K 2023-03-22 17:55:44,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=6.71 KB/6870 for 2343ed3ff2707ca3be5523fce8688282 in 1659ms, sequenceid=640, compaction requested=false 2023-03-22 17:55:44,591 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:46,099 DEBUG [Thread-2383] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:55285 2023-03-22 17:55:46,099 DEBUG [Thread-2383] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:49,168 DEBUG [Thread-2381] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:55285 2023-03-22 17:55:49,168 DEBUG [Thread-2381] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 236 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 56 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 230 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1599 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 4797 rows 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1605 2023-03-22 17:55:49,168 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 4815 rows 2023-03-22 17:55:49,169 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1602 2023-03-22 17:55:49,169 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 4806 rows 2023-03-22 17:55:49,169 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1599 2023-03-22 17:55:49,169 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 4797 rows 2023-03-22 17:55:49,169 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1603 2023-03-22 17:55:49,169 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 4809 rows 2023-03-22 17:55:49,169 INFO [Listener at localhost/44491] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-22 17:55:49,169 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b77d6d7 to 127.0.0.1:55285 2023-03-22 17:55:49,169 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:55:49,170 INFO [Listener at localhost/44491] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-22 17:55:49,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-22 17:55:49,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:49,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-22 17:55:49,175 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507749175"}]},"ts":"1679507749175"} 2023-03-22 17:55:49,176 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-22 17:55:49,178 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-22 17:55:49,178 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, UNASSIGN}] 2023-03-22 17:55:49,179 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, UNASSIGN 2023-03-22 17:55:49,180 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=2343ed3ff2707ca3be5523fce8688282, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:49,180 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507749180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507749180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507749180"}]},"ts":"1679507749180"} 2023-03-22 17:55:49,181 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:55:49,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-22 17:55:49,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:49,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2343ed3ff2707ca3be5523fce8688282, disabling compactions & flushes 2023-03-22 17:55:49,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:49,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:49,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. after waiting 0 ms 2023-03-22 17:55:49,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:49,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2343ed3ff2707ca3be5523fce8688282 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-22 17:55:49,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=A 2023-03-22 17:55:49,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:49,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=B 2023-03-22 17:55:49,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:49,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2343ed3ff2707ca3be5523fce8688282, store=C 2023-03-22 17:55:49,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:49,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-22 17:55:49,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032298b68000749a42758a2896e2c84e8484 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032298b68000749a42758a2896e2c84e8484 2023-03-22 17:55:49,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-22 17:55:50,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=648, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/69ab7d6654e34a8caaedb499c60df3ef 2023-03-22 17:55:50,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-22 17:55:50,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/f0d55109068b4a669f49a62538d24fd0 2023-03-22 17:55:50,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/02171d9f71dd4f60bfb441eb66dabdd8 2023-03-22 17:55:50,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/A/69ab7d6654e34a8caaedb499c60df3ef as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/69ab7d6654e34a8caaedb499c60df3ef 2023-03-22 17:55:50,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/69ab7d6654e34a8caaedb499c60df3ef, entries=100, sequenceid=648, filesize=18.6 K 2023-03-22 17:55:50,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/B/f0d55109068b4a669f49a62538d24fd0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/f0d55109068b4a669f49a62538d24fd0 2023-03-22 17:55:50,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/f0d55109068b4a669f49a62538d24fd0, entries=100, sequenceid=648, filesize=9.5 K 2023-03-22 17:55:50,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/.tmp/C/02171d9f71dd4f60bfb441eb66dabdd8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/02171d9f71dd4f60bfb441eb66dabdd8 2023-03-22 17:55:50,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/02171d9f71dd4f60bfb441eb66dabdd8, entries=100, sequenceid=648, filesize=9.5 K 2023-03-22 17:55:50,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 2343ed3ff2707ca3be5523fce8688282 in 1645ms, sequenceid=648, compaction requested=true 2023-03-22 17:55:50,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5672101319c44490bf9c407c04f131c5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/490b333659494e9ca56d95edea1e8715, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/8faa83002fa04797872fe0ede6bc4db3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2eab2e03eafd41ad8898589d3a5c4319, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b182e24c3f794e9a8bc76822449d7353] to archive 2023-03-22 17:55:50,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:50,979 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5672101319c44490bf9c407c04f131c5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/5672101319c44490bf9c407c04f131c5 2023-03-22 17:55:50,979 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/490b333659494e9ca56d95edea1e8715 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/490b333659494e9ca56d95edea1e8715 2023-03-22 17:55:50,980 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/8faa83002fa04797872fe0ede6bc4db3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/8faa83002fa04797872fe0ede6bc4db3 2023-03-22 17:55:50,981 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2eab2e03eafd41ad8898589d3a5c4319 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/2eab2e03eafd41ad8898589d3a5c4319 2023-03-22 17:55:50,982 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b182e24c3f794e9a8bc76822449d7353 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/b182e24c3f794e9a8bc76822449d7353 2023-03-22 17:55:50,983 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b36976dfa5644a5abe3ff420e974aded, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/09db8b5ca9904d92a9fcff38b6edc9c8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7b5d9206d2bf4c3c9e788a6d7697aff9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/e0ee8b296f704016954a995b20f84f33, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/131642d093404c2c91d7212bfa2f9ad7] to archive 2023-03-22 17:55:50,983 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:50,984 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b36976dfa5644a5abe3ff420e974aded to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/b36976dfa5644a5abe3ff420e974aded 2023-03-22 17:55:50,985 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/09db8b5ca9904d92a9fcff38b6edc9c8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/09db8b5ca9904d92a9fcff38b6edc9c8 2023-03-22 17:55:50,986 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7b5d9206d2bf4c3c9e788a6d7697aff9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/7b5d9206d2bf4c3c9e788a6d7697aff9 2023-03-22 17:55:50,987 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/e0ee8b296f704016954a995b20f84f33 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/e0ee8b296f704016954a995b20f84f33 2023-03-22 17:55:50,988 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/131642d093404c2c91d7212bfa2f9ad7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/131642d093404c2c91d7212bfa2f9ad7 2023-03-22 17:55:50,989 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/47b32e7ee8da4f43b384c4d31a9f6402, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/0ce43f8443dc4cafb1724b355d2d65fe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/56c3213436a94acda0a4c2bb039092ff, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d7540f3c8b804c50b4fe3389261c84e1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d9c3a8c18e804e17ae83f1d7d9f2ad18] to archive 2023-03-22 17:55:50,989 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:55:50,990 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/47b32e7ee8da4f43b384c4d31a9f6402 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/47b32e7ee8da4f43b384c4d31a9f6402 2023-03-22 17:55:50,991 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/0ce43f8443dc4cafb1724b355d2d65fe to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/0ce43f8443dc4cafb1724b355d2d65fe 2023-03-22 17:55:50,992 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/56c3213436a94acda0a4c2bb039092ff to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/56c3213436a94acda0a4c2bb039092ff 2023-03-22 17:55:50,993 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d7540f3c8b804c50b4fe3389261c84e1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d7540f3c8b804c50b4fe3389261c84e1 2023-03-22 17:55:50,994 DEBUG [StoreCloser-TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d9c3a8c18e804e17ae83f1d7d9f2ad18 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/d9c3a8c18e804e17ae83f1d7d9f2ad18 2023-03-22 17:55:50,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/recovered.edits/651.seqid, newMaxSeqId=651, maxSeqId=4 2023-03-22 17:55:50,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282. 2023-03-22 17:55:50,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2343ed3ff2707ca3be5523fce8688282: 2023-03-22 17:55:50,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:51,000 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=2343ed3ff2707ca3be5523fce8688282, regionState=CLOSED 2023-03-22 17:55:51,000 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507751000"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507751000"}]},"ts":"1679507751000"} 2023-03-22 17:55:51,002 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-22 17:55:51,002 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 2343ed3ff2707ca3be5523fce8688282, server=jenkins-hbase4.apache.org,34403,1679507617723 in 1.8200 sec 2023-03-22 17:55:51,003 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-22 17:55:51,003 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2343ed3ff2707ca3be5523fce8688282, UNASSIGN in 1.8240 sec 2023-03-22 17:55:51,003 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507751003"}]},"ts":"1679507751003"} 2023-03-22 17:55:51,004 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-22 17:55:51,010 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-22 17:55:51,011 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8390 sec 2023-03-22 17:55:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-22 17:55:51,279 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-22 17:55:51,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-22 17:55:51,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:51,282 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:51,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-22 17:55:51,282 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:51,286 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:51,288 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/recovered.edits] 2023-03-22 17:55:51,290 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/4e2c7ab4b150463497b313125a92a161 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/4e2c7ab4b150463497b313125a92a161 2023-03-22 17:55:51,291 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/69ab7d6654e34a8caaedb499c60df3ef to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/69ab7d6654e34a8caaedb499c60df3ef 2023-03-22 17:55:51,292 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a9cc3668fad34b1690e1436394a077a5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/A/a9cc3668fad34b1690e1436394a077a5 2023-03-22 17:55:51,294 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/2d9acd43cb3e4e97b508d4376ad8a23f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/2d9acd43cb3e4e97b508d4376ad8a23f 2023-03-22 17:55:51,295 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d0a542196c114e9982ba15f0942a4ce2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/d0a542196c114e9982ba15f0942a4ce2 2023-03-22 17:55:51,296 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/f0d55109068b4a669f49a62538d24fd0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/B/f0d55109068b4a669f49a62538d24fd0 2023-03-22 17:55:51,298 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/02171d9f71dd4f60bfb441eb66dabdd8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/02171d9f71dd4f60bfb441eb66dabdd8 2023-03-22 17:55:51,299 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4afd02401262421b830e75e2c538e7cb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/4afd02401262421b830e75e2c538e7cb 2023-03-22 17:55:51,300 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bf5447d8a0bd43cb8ef4e5f50b2af746 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/C/bf5447d8a0bd43cb8ef4e5f50b2af746 2023-03-22 17:55:51,302 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/recovered.edits/651.seqid to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282/recovered.edits/651.seqid 2023-03-22 17:55:51,302 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/2343ed3ff2707ca3be5523fce8688282 2023-03-22 17:55:51,302 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:55:51,303 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-22 17:55:51,303 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-22 17:55:51,305 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322107846fc1e6e4e708a6e37a8f0db8a0c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322107846fc1e6e4e708a6e37a8f0db8a0c 2023-03-22 17:55:51,306 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032212e3888d041c49ae8953202c1902f244 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032212e3888d041c49ae8953202c1902f244 2023-03-22 17:55:51,307 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303221f28bc78bc944a94bb7d664b3cf4f000 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303221f28bc78bc944a94bb7d664b3cf4f000 2023-03-22 17:55:51,308 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303221fa48b288ffb4a269d6b6abdaede148c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303221fa48b288ffb4a269d6b6abdaede148c 2023-03-22 17:55:51,309 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303222a1a5fee9933441e84ba7d635e648b7b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303222a1a5fee9933441e84ba7d635e648b7b 2023-03-22 17:55:51,310 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322308b8414d6c74b43a59169ba1dd50e08 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322308b8414d6c74b43a59169ba1dd50e08 2023-03-22 17:55:51,311 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223236e93d240d459480d49501450c9b22 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223236e93d240d459480d49501450c9b22 2023-03-22 17:55:51,312 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223b966c63a75347119ce2a578642466af to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303223b966c63a75347119ce2a578642466af 2023-03-22 17:55:51,313 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322464fe4fbb5064c2a9e2ab1815a714fb7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322464fe4fbb5064c2a9e2ab1815a714fb7 2023-03-22 17:55:51,314 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303225ca7c6d3c39e49feaf973e0861d099c7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303225ca7c6d3c39e49feaf973e0861d099c7 2023-03-22 17:55:51,315 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303226ab3e1f0aab54e36befff4b9c096c787 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303226ab3e1f0aab54e36befff4b9c096c787 2023-03-22 17:55:51,315 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322869447db811c4e2ba0c1f271dbfd8fad to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322869447db811c4e2ba0c1f271dbfd8fad 2023-03-22 17:55:51,316 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322961ca70dce9f459289e53d48940dd745 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322961ca70dce9f459289e53d48940dd745 2023-03-22 17:55:51,317 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032298b68000749a42758a2896e2c84e8484 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032298b68000749a42758a2896e2c84e8484 2023-03-22 17:55:51,318 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303229d879fd52b1e47ebb3730b6402c57a28 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303229d879fd52b1e47ebb3730b6402c57a28 2023-03-22 17:55:51,318 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a23a2b306de84f6992ba69835843eb1a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a23a2b306de84f6992ba69835843eb1a 2023-03-22 17:55:51,319 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a715049a82e743e4bc381915a5d900bc to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a715049a82e743e4bc381915a5d900bc 2023-03-22 17:55:51,320 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c785aac699884547a452563cf6b8f7d5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c785aac699884547a452563cf6b8f7d5 2023-03-22 17:55:51,321 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322cc07dbb9551743e9bba4853407df7123 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322cc07dbb9551743e9bba4853407df7123 2023-03-22 17:55:51,321 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322de21c052aa0540c7a152d873def560dd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322de21c052aa0540c7a152d873def560dd 2023-03-22 17:55:51,322 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-22 17:55:51,324 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:51,325 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-22 17:55:51,326 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-22 17:55:51,327 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:51,327 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-22 17:55:51,327 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679507751327"}]},"ts":"9223372036854775807"} 2023-03-22 17:55:51,328 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-22 17:55:51,329 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2343ed3ff2707ca3be5523fce8688282, NAME => 'TestAcidGuarantees,,1679507719544.2343ed3ff2707ca3be5523fce8688282.', STARTKEY => '', ENDKEY => ''}] 2023-03-22 17:55:51,329 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-22 17:55:51,329 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679507751329"}]},"ts":"9223372036854775807"} 2023-03-22 17:55:51,330 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-22 17:55:51,336 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:51,336 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-03-22 17:55:51,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-22 17:55:51,383 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-22 17:55:51,394 INFO [Listener at localhost/44491] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=528 (was 524) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=308 (was 283) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=5852 (was 5876) 2023-03-22 17:55:51,405 INFO [Listener at localhost/44491] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=226, OpenFileDescriptor=528, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=169, AvailableMemoryMB=5852 2023-03-22 17:55:51,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:55:51,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:55:51,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-22 17:55:51,409 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-22 17:55:51,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-22 17:55:51,409 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-22 17:55:51,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-22 17:55:51,410 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:51,411 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282 empty. 2023-03-22 17:55:51,411 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:51,411 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:55:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-22 17:55:51,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-22 17:55:51,818 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:55:51,819 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 523fd4cd7256d4e27c9367c2949e4282, NAME => 'TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp 2023-03-22 17:55:52,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-22 17:55:52,230 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:55:52,230 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 523fd4cd7256d4e27c9367c2949e4282, disabling compactions & flushes 2023-03-22 17:55:52,230 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:52,230 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:52,230 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. after waiting 0 ms 2023-03-22 17:55:52,230 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:52,230 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:52,230 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:52,232 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-22 17:55:52,233 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507752233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507752233"}]},"ts":"1679507752233"} 2023-03-22 17:55:52,234 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-22 17:55:52,234 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-22 17:55:52,234 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507752234"}]},"ts":"1679507752234"} 2023-03-22 17:55:52,235 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-22 17:55:52,238 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=523fd4cd7256d4e27c9367c2949e4282, ASSIGN}] 2023-03-22 17:55:52,239 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=523fd4cd7256d4e27c9367c2949e4282, ASSIGN 2023-03-22 17:55:52,240 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=523fd4cd7256d4e27c9367c2949e4282, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=false 2023-03-22 17:55:52,391 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=523fd4cd7256d4e27c9367c2949e4282, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:52,391 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507752391"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507752391"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507752391"}]},"ts":"1679507752391"} 2023-03-22 17:55:52,392 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:55:52,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-22 17:55:52,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:52,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 523fd4cd7256d4e27c9367c2949e4282, NAME => 'TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:55:52,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:55:52,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,551 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,552 DEBUG [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A 2023-03-22 17:55:52,552 DEBUG [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A 2023-03-22 17:55:52,552 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:52,553 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 523fd4cd7256d4e27c9367c2949e4282 columnFamilyName A 2023-03-22 17:55:52,553 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.HStore(310): Store=523fd4cd7256d4e27c9367c2949e4282/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:52,553 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,555 DEBUG [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B 2023-03-22 17:55:52,555 DEBUG [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B 2023-03-22 17:55:52,555 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:52,555 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 523fd4cd7256d4e27c9367c2949e4282 columnFamilyName B 2023-03-22 17:55:52,556 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.HStore(310): Store=523fd4cd7256d4e27c9367c2949e4282/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:52,556 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,557 DEBUG [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C 2023-03-22 17:55:52,557 DEBUG [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C 2023-03-22 17:55:52,557 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:55:52,557 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 523fd4cd7256d4e27c9367c2949e4282 columnFamilyName C 2023-03-22 17:55:52,558 INFO [StoreOpener-523fd4cd7256d4e27c9367c2949e4282-1] regionserver.HStore(310): Store=523fd4cd7256d4e27c9367c2949e4282/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:55:52,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:52,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:55:52,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:52,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:55:52,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 523fd4cd7256d4e27c9367c2949e4282; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75199590, jitterRate=0.12056121230125427}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:55:52,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:52,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., pid=49, masterSystemTime=1679507752544 2023-03-22 17:55:52,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:52,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:52,566 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=523fd4cd7256d4e27c9367c2949e4282, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:52,566 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507752566"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507752566"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507752566"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507752566"}]},"ts":"1679507752566"} 2023-03-22 17:55:52,568 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-22 17:55:52,568 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 in 175 msec 2023-03-22 17:55:52,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-22 17:55:52,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=523fd4cd7256d4e27c9367c2949e4282, ASSIGN in 330 msec 2023-03-22 17:55:52,569 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-22 17:55:52,569 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507752569"}]},"ts":"1679507752569"} 2023-03-22 17:55:52,570 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-22 17:55:52,572 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-22 17:55:52,573 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-03-22 17:55:53,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-22 17:55:53,514 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-22 17:55:53,514 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x3a29ce5a to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,521 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73eb7080, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,521 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,523 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54932, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,524 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-22 17:55:53,526 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54988, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-22 17:55:53,527 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x5e372785 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,530 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a987907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,530 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x501c5a88 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,535 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37705401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,535 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x27f2d997 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,540 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15880ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,540 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,545 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,545 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,548 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,549 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,554 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,554 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,561 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,562 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,566 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,567 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,570 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,570 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:55:53,573 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:55:53,578 DEBUG [hconnection-0x3a812d7b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,578 DEBUG [hconnection-0x4acea22f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,579 DEBUG [hconnection-0x8684a7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,579 DEBUG [hconnection-0x178113fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,579 DEBUG [hconnection-0x61511f3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,579 DEBUG [hconnection-0xe088143-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,580 DEBUG [hconnection-0xa5a7a8f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,580 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54954, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,580 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54968, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,584 DEBUG [hconnection-0x3d5da49e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,584 DEBUG [hconnection-0x7a231dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,585 DEBUG [hconnection-0x14f09f19-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:55:53,587 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54956, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,587 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54976, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:53,588 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54942, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,588 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54970, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,588 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54980, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,588 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54996, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,591 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55012, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,592 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:55:53,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:53,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:53,601 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:53,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:53,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:53,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:53,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:53,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:55:53,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:53,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:53,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:53,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:53,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:53,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:53,605 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,605 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:53,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:53,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,607 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:53,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:53,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:53,620 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:53,620 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:53,620 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:53,620 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:53,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:53,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507813622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507753616, max timestamp=1679507753623Num uniques 859; 2023-03-22 17:55:53,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:53,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507753616, max timestamp=1679507753623Num uniques 859; 2023-03-22 17:55:53,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507753616, max timestamp=1679507753623Num uniques 859; 2023-03-22 17:55:53,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:53,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:55:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d52b1c6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:53,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:53,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,627 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,627 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:53,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:53,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:53,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:53,632 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:53,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:53,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:53,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:53,632 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:53,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,633 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:53,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:53,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:53,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507813634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507813634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,636 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/4fdbfd30ea0544cb8b10c194edfb113f 2023-03-22 17:55:53,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:53,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:53,639 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:53,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:53,640 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:53,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:53,640 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,640 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,640 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,640 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:53,640 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:53,640 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:53,640 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507813635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507813635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507813726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,740 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:53,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507813736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:53,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507813736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:53,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507813743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507813744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:53,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:53,748 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:53,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:53,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:53,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:53,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,749 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:53,749 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:53,750 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,750 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:53,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:53,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:53,753 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:53,753 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:53,753 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:53,753 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:53,753 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:53,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:53,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:53,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65312db8[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:53,756 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,758 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:53,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:53,758 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:53,758 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:53,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:53,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:53,760 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:53,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:53,762 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:53,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:53,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:53,763 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:53,763 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:53,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:53,764 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,765 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:53,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,765 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:53,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:53,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:53,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:53,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:53,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:53,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:53,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:53,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:53,768 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:53,768 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:53,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,769 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,868 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:53,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:53,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:53,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:53,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:53,875 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:53,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:53,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:53,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:53,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:53,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:53,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:53,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:53,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:53,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:53,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:53,884 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:53,898 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,898 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:53,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:53,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40d637cc[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:53,901 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:53,901 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,903 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:53,904 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:53,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:53,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:53,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:53,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:53,905 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:53,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:53,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:53,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:53,905 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:53,907 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,907 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:53,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,907 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:53,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:53,908 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,913 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:53,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:53,915 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:53,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:53,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:53,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:53,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:53,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,915 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:53,916 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:53,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,916 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:53,916 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:53,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:53,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:53,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507813934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507813942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507813943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507813947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:53,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:53,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507813947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,016 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:54,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:54,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:54,019 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:54,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:54,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:54,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,019 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,021 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:54,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:54,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:54,023 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,023 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:54,023 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:54,023 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:54,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:54,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f087609[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:54,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:54,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:54,029 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,029 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:54,030 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:54,030 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:54,030 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,030 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:54,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:54,033 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:54,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:54,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:54,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:54,033 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:54,035 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,035 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,035 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:54,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,036 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,038 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:54,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:54,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:54,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:54,039 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:54,039 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:54,039 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,039 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:54,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:54,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/b28bc1bf2f444391a0373c17b010aaa0 2023-03-22 17:55:54,139 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:54,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:54,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:54,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:54,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:54,142 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:54,142 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,142 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:54,142 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:54,142 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,142 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,145 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,145 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,145 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,145 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:54,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:54,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:54,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:54,146 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,146 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:54,147 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:54,148 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,148 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc3eaf[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:54,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:54,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,151 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:54,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:54,152 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:54,156 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:54,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:54,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:54,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:54,156 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:54,162 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,162 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,162 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,163 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,164 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:54,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,166 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:54,166 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,166 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:54,166 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:54,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507814240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507814250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507814250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507814253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507814255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,267 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:54,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:54,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:54,269 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:54,269 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,269 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:54,269 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:54,270 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,270 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,271 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:54,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:54,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:54,273 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,273 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:54,273 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:54,273 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:54,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:54,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,275 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:54,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b4c9f05[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:54,277 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:54,277 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,279 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:54,279 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,279 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:54,280 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:54,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:54,282 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:54,282 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:54,282 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:54,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:54,283 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:54,286 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,286 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,286 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,288 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,293 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,293 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,293 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:54,293 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:54,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:54,293 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,294 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,294 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:54,294 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:54,294 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,294 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,294 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,294 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:54,394 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:54,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:54,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:54,397 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:54,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:54,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:54,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,399 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,399 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,399 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,399 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:54,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:54,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:54,400 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,400 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:54,400 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:54,400 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:54,402 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:54,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a55adf7[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:54,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:54,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,405 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,405 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:54,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:54,405 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:54,405 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:54,407 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:54,409 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:54,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:54,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:54,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:54,409 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:54,411 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,411 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,411 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:54,411 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:54,412 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,415 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:54,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:54,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:54,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:54,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:54,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:54,415 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:54,415 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:54,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/c4a1d79cce1d465792063315973694f5 2023-03-22 17:55:54,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/4fdbfd30ea0544cb8b10c194edfb113f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/4fdbfd30ea0544cb8b10c194edfb113f 2023-03-22 17:55:54,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/4fdbfd30ea0544cb8b10c194edfb113f, entries=200, sequenceid=12, filesize=13.9 K 2023-03-22 17:55:54,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/b28bc1bf2f444391a0373c17b010aaa0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/b28bc1bf2f444391a0373c17b010aaa0 2023-03-22 17:55:54,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/b28bc1bf2f444391a0373c17b010aaa0, entries=150, sequenceid=12, filesize=11.6 K 2023-03-22 17:55:54,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/c4a1d79cce1d465792063315973694f5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c4a1d79cce1d465792063315973694f5 2023-03-22 17:55:54,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c4a1d79cce1d465792063315973694f5, entries=150, sequenceid=12, filesize=11.6 K 2023-03-22 17:55:54,474 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 871ms, sequenceid=12, compaction requested=false 2023-03-22 17:55:54,474 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-22 17:55:54,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:54,516 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:54,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:54,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:54,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:54,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:54,519 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:54,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:54,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:54,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,521 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,521 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:54,521 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,521 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:54,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:54,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:54,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:54,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:54,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:54,524 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-22 17:55:54,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:54,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:54,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:54,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:54,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:54,524 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:55:54,532 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/8fb638a6cf004d77acb03f9102d1bef9 2023-03-22 17:55:54,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:54,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:54,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507814758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507814761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507814761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507814762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507814765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507814866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507814867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507814868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507814874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:54,948 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/52cbe939246e47948e69113de2b35c02 2023-03-22 17:55:55,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507815070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507815072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507815074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507815079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,358 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/5ca93ffdaa914f83a60afdb71f66ad9c 2023-03-22 17:55:55,362 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/8fb638a6cf004d77acb03f9102d1bef9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fb638a6cf004d77acb03f9102d1bef9 2023-03-22 17:55:55,369 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fb638a6cf004d77acb03f9102d1bef9, entries=150, sequenceid=49, filesize=11.6 K 2023-03-22 17:55:55,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/52cbe939246e47948e69113de2b35c02 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/52cbe939246e47948e69113de2b35c02 2023-03-22 17:55:55,374 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/52cbe939246e47948e69113de2b35c02, entries=150, sequenceid=49, filesize=11.6 K 2023-03-22 17:55:55,375 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/5ca93ffdaa914f83a60afdb71f66ad9c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5ca93ffdaa914f83a60afdb71f66ad9c 2023-03-22 17:55:55,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507815371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,381 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5ca93ffdaa914f83a60afdb71f66ad9c, entries=150, sequenceid=49, filesize=11.6 K 2023-03-22 17:55:55,382 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 523fd4cd7256d4e27c9367c2949e4282 in 858ms, sequenceid=49, compaction requested=false 2023-03-22 17:55:55,385 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:55,385 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:55,385 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:55,387 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,388 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:55:55,388 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,388 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7601288a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:55,390 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:55,390 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,392 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:55,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:55,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:55,404 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:55,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:55,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:55,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:55,404 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:55,406 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,406 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:55,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,406 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:55,407 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,409 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/32f266bed58744b88ff9b073554e14b6 2023-03-22 17:55:55,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:55,412 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:55,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:55,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:55,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:55,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:55,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,413 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:55,413 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:55,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:55,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:55,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/0795b7a5d39e42b2a4cacc02c19cc0de 2023-03-22 17:55:55,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507815426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/6d034e6aabad47958089f784b4fb7b6c 2023-03-22 17:55:55,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507815429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/32f266bed58744b88ff9b073554e14b6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/32f266bed58744b88ff9b073554e14b6 2023-03-22 17:55:55,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/32f266bed58744b88ff9b073554e14b6, entries=250, sequenceid=61, filesize=16.2 K 2023-03-22 17:55:55,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/0795b7a5d39e42b2a4cacc02c19cc0de as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0795b7a5d39e42b2a4cacc02c19cc0de 2023-03-22 17:55:55,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507815438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0795b7a5d39e42b2a4cacc02c19cc0de, entries=150, sequenceid=61, filesize=11.6 K 2023-03-22 17:55:55,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/6d034e6aabad47958089f784b4fb7b6c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/6d034e6aabad47958089f784b4fb7b6c 2023-03-22 17:55:55,456 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/6d034e6aabad47958089f784b4fb7b6c, entries=150, sequenceid=61, filesize=11.6 K 2023-03-22 17:55:55,456 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 523fd4cd7256d4e27c9367c2949e4282 in 68ms, sequenceid=61, compaction requested=true 2023-03-22 17:55:55,456 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:55,456 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:55,457 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:55,457 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:55,457 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:55,457 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:55,458 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:55,458 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:55:55,458 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,458 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/b28bc1bf2f444391a0373c17b010aaa0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/52cbe939246e47948e69113de2b35c02, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0795b7a5d39e42b2a4cacc02c19cc0de] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=34.8 K 2023-03-22 17:55:55,459 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b28bc1bf2f444391a0373c17b010aaa0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679507753598 2023-03-22 17:55:55,459 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:55,459 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:55:55,459 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,460 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/4fdbfd30ea0544cb8b10c194edfb113f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fb638a6cf004d77acb03f9102d1bef9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/32f266bed58744b88ff9b073554e14b6] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=41.7 K 2023-03-22 17:55:55,460 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 4fdbfd30ea0544cb8b10c194edfb113f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679507753598 2023-03-22 17:55:55,460 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 52cbe939246e47948e69113de2b35c02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679507753630 2023-03-22 17:55:55,460 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 8fb638a6cf004d77acb03f9102d1bef9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679507753630 2023-03-22 17:55:55,461 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0795b7a5d39e42b2a4cacc02c19cc0de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507754757 2023-03-22 17:55:55,461 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 32f266bed58744b88ff9b073554e14b6, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507754757 2023-03-22 17:55:55,466 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#384 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:55,467 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#385 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:55,513 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:55,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:55,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:55,516 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:55,516 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:55,516 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:55,516 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:55,517 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,517 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,518 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,518 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:55,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:55,518 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,518 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:55,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:55,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:55,520 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,520 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:55,520 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-22 17:55:55,521 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:55,521 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,521 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:55,521 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,521 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:55,521 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,537 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/453528ac90d7472995fcdeb6183e74c9 2023-03-22 17:55:55,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:55,550 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:55,553 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/38fce18265604c64a8c4efc3c467890b 2023-03-22 17:55:55,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507815560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507815561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,568 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/d5e6586101754eaabec80e2e8b2b209a 2023-03-22 17:55:55,574 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/453528ac90d7472995fcdeb6183e74c9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/453528ac90d7472995fcdeb6183e74c9 2023-03-22 17:55:55,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507815566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,578 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/453528ac90d7472995fcdeb6183e74c9, entries=150, sequenceid=81, filesize=11.6 K 2023-03-22 17:55:55,579 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/38fce18265604c64a8c4efc3c467890b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/38fce18265604c64a8c4efc3c467890b 2023-03-22 17:55:55,586 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/38fce18265604c64a8c4efc3c467890b, entries=150, sequenceid=81, filesize=11.6 K 2023-03-22 17:55:55,586 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/d5e6586101754eaabec80e2e8b2b209a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d5e6586101754eaabec80e2e8b2b209a 2023-03-22 17:55:55,589 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d5e6586101754eaabec80e2e8b2b209a, entries=150, sequenceid=81, filesize=11.6 K 2023-03-22 17:55:55,590 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 523fd4cd7256d4e27c9367c2949e4282 in 70ms, sequenceid=81, compaction requested=true 2023-03-22 17:55:55,590 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:55,590 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:55,592 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,593 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:55,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106e79a3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:55,594 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,595 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:55,595 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,595 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:55,598 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:55,602 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:55,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:55,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:55,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:55,602 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:55,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:55,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:55,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,605 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,610 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:55,610 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:55,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:55,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:55,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:55,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:55,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:55,611 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:55,611 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:55,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:55,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:55,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-22 17:55:55,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:55,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:55,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:55,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:55,711 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:55,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:55,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:55,715 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:55,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:55,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:55,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:55,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:55,717 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:55,718 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,718 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:55,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:55,721 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:55,721 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:55,721 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:55,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,722 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507815712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,722 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:55,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:55,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:55,724 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:55,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,726 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:55,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd5681c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:55,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:55,728 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507815720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,730 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:55,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,730 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:55,733 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507815722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:55,736 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:55,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:55,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:55,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:55,736 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:55,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:55,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:55,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:55,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:55,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:55,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:55,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:55,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:55,743 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:55,743 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:55,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507815772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507815825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507815829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507815836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,843 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:55,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:55,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:55,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:55,846 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:55,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:55,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:55,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:55,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,854 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:55,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:55,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,854 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:55,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:55,856 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,856 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:55,856 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:55,856 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:55,858 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,858 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:55,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28585404[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:55,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:55,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:55,863 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,863 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:55,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:55,868 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:55,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:55,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:55,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:55,868 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:55,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:55,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,872 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:55,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:55,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:55,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:55,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:55,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:55,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:55,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,877 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:55,877 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:55,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507815879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,884 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/cb1cb9f2ebf64154865620cc13d4302f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/cb1cb9f2ebf64154865620cc13d4302f 2023-03-22 17:55:55,887 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/94f3a541d4fc46d6b10d81a82d9c488f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/94f3a541d4fc46d6b10d81a82d9c488f 2023-03-22 17:55:55,891 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into cb1cb9f2ebf64154865620cc13d4302f(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:55,891 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:55,891 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=13, startTime=1679507755456; duration=0sec 2023-03-22 17:55:55,891 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:55,891 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:55:55,894 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:55:55,894 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:55:55,894 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,894 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c4a1d79cce1d465792063315973694f5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5ca93ffdaa914f83a60afdb71f66ad9c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/6d034e6aabad47958089f784b4fb7b6c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d5e6586101754eaabec80e2e8b2b209a] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=46.4 K 2023-03-22 17:55:55,894 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c4a1d79cce1d465792063315973694f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679507753598 2023-03-22 17:55:55,895 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into 94f3a541d4fc46d6b10d81a82d9c488f(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:55,895 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:55,895 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=13, startTime=1679507755457; duration=0sec 2023-03-22 17:55:55,896 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:55,896 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5ca93ffdaa914f83a60afdb71f66ad9c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679507753630 2023-03-22 17:55:55,896 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 6d034e6aabad47958089f784b4fb7b6c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507754757 2023-03-22 17:55:55,896 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d5e6586101754eaabec80e2e8b2b209a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679507755425 2023-03-22 17:55:55,907 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#390 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:55,938 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/bb9b0c4f73e445819a51a8618c4325ab as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb9b0c4f73e445819a51a8618c4325ab 2023-03-22 17:55:55,945 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into bb9b0c4f73e445819a51a8618c4325ab(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:55,945 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:55,945 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=12, startTime=1679507755457; duration=0sec 2023-03-22 17:55:55,945 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:55,977 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:55,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:55,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:55,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:55,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:55,986 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:55,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:55,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:55,986 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:55,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,994 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:55,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,994 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:55,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:55,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:55,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:55,996 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,996 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:55,996 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:55,996 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:55,998 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57e26cfd[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:56,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:56,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,001 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:56,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:56,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:56,001 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,001 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:56,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:56,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:56,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:56,003 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:56,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:56,026 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:56,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:56,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:56,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:56,026 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:56,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:56,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:56,030 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507816028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,037 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,038 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:56,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:56,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:56,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:56,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:56,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,038 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:56,038 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:56,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507816037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:56,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507816044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/b9ace1850c7d4e0dbf1036aea1bc72a8 2023-03-22 17:55:56,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/8b9b0a588399477caf34e478e69019e0 2023-03-22 17:55:56,138 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:56,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:56,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:56,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:56,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:56,156 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:56,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:56,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:56,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:56,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,157 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,158 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,158 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,158 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:56,158 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:56,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ca6c77931d5b45d2920917c762ea5779 2023-03-22 17:55:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:56,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/b9ace1850c7d4e0dbf1036aea1bc72a8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b9ace1850c7d4e0dbf1036aea1bc72a8 2023-03-22 17:55:56,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:56,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:56,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b9ace1850c7d4e0dbf1036aea1bc72a8, entries=200, sequenceid=94, filesize=13.9 K 2023-03-22 17:55:56,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/8b9b0a588399477caf34e478e69019e0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/8b9b0a588399477caf34e478e69019e0 2023-03-22 17:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:56,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:56,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:56,176 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:56,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/8b9b0a588399477caf34e478e69019e0, entries=150, sequenceid=94, filesize=11.6 K 2023-03-22 17:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ca6c77931d5b45d2920917c762ea5779 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca6c77931d5b45d2920917c762ea5779 2023-03-22 17:55:56,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:56,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a2b4c3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:56,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:56,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:56,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:56,187 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:56,190 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:56,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:56,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:56,190 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:56,190 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:56,193 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:56,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:56,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:56,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:56,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:56,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,200 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:56,200 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:56,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:56,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca6c77931d5b45d2920917c762ea5779, entries=150, sequenceid=94, filesize=11.6 K 2023-03-22 17:55:56,202 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 523fd4cd7256d4e27c9367c2949e4282 in 529ms, sequenceid=94, compaction requested=true 2023-03-22 17:55:56,202 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:56,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:56,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:56,202 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:56,202 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:56,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:56,212 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:56,212 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:55:56,212 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:56,212 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,212 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:55:56,212 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/94f3a541d4fc46d6b10d81a82d9c488f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/38fce18265604c64a8c4efc3c467890b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/8b9b0a588399477caf34e478e69019e0] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=34.9 K 2023-03-22 17:55:56,212 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,213 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/cb1cb9f2ebf64154865620cc13d4302f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/453528ac90d7472995fcdeb6183e74c9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b9ace1850c7d4e0dbf1036aea1bc72a8] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=37.2 K 2023-03-22 17:55:56,216 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 94f3a541d4fc46d6b10d81a82d9c488f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507754757 2023-03-22 17:55:56,217 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 38fce18265604c64a8c4efc3c467890b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679507755425 2023-03-22 17:55:56,218 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting cb1cb9f2ebf64154865620cc13d4302f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679507754757 2023-03-22 17:55:56,218 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 8b9b0a588399477caf34e478e69019e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679507755559 2023-03-22 17:55:56,218 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 453528ac90d7472995fcdeb6183e74c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679507755425 2023-03-22 17:55:56,219 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b9ace1850c7d4e0dbf1036aea1bc72a8, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679507755559 2023-03-22 17:55:56,237 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#393 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:56,246 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#394 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:56,274 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/fd9ab0d76d334cde9866f6b19822d9dc as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fd9ab0d76d334cde9866f6b19822d9dc 2023-03-22 17:55:56,277 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into fd9ab0d76d334cde9866f6b19822d9dc(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:56,277 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:56,277 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=13, startTime=1679507756202; duration=0sec 2023-03-22 17:55:56,277 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:56,277 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-22 17:55:56,278 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-22 17:55:56,278 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-22 17:55:56,278 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. because compaction request was cancelled 2023-03-22 17:55:56,301 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:56,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:56,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:56,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:56,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:56,307 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:56,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:56,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:56,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:56,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,308 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,309 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:56,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:56,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:56,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:56,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:56,312 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-22 17:55:56,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:56,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:56,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:56,312 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:56,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:56,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507816367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507816368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507816368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507816474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507816474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507816474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,537 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-22 17:55:56,666 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/1b98d5acb4ed40d1a7795659aeb46c99 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1b98d5acb4ed40d1a7795659aeb46c99 2023-03-22 17:55:56,670 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into 1b98d5acb4ed40d1a7795659aeb46c99(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:56,670 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:56,670 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=13, startTime=1679507756202; duration=0sec 2023-03-22 17:55:56,670 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:56,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507816676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507816677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:56,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507816688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,733 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/0cdc37ce18154e18999999e956cd2a1c 2023-03-22 17:55:56,742 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/c4aa12ec770b4b339aa2028ab4cdab07 2023-03-22 17:55:56,751 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/320b138d0af3499288c641226014f2ec 2023-03-22 17:55:56,754 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/0cdc37ce18154e18999999e956cd2a1c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/0cdc37ce18154e18999999e956cd2a1c 2023-03-22 17:55:56,759 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/0cdc37ce18154e18999999e956cd2a1c, entries=150, sequenceid=117, filesize=11.6 K 2023-03-22 17:55:56,759 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/c4aa12ec770b4b339aa2028ab4cdab07 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4aa12ec770b4b339aa2028ab4cdab07 2023-03-22 17:55:56,762 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4aa12ec770b4b339aa2028ab4cdab07, entries=150, sequenceid=117, filesize=11.6 K 2023-03-22 17:55:56,763 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/320b138d0af3499288c641226014f2ec as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/320b138d0af3499288c641226014f2ec 2023-03-22 17:55:56,765 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/320b138d0af3499288c641226014f2ec, entries=150, sequenceid=117, filesize=11.6 K 2023-03-22 17:55:56,766 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 523fd4cd7256d4e27c9367c2949e4282 in 454ms, sequenceid=117, compaction requested=true 2023-03-22 17:55:56,766 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:56,766 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,766 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:56,766 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:56,766 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,766 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:56,766 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,768 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:56,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3afc3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:56,771 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:56,771 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:56,772 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,772 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:56,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:56,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:56,780 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:56,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:56,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:56,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:56,780 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:56,782 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,782 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,782 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,784 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,788 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:56,788 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:56,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:56,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:56,788 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:56,788 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:56,788 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:56,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:56,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:56,888 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:56,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:56,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:56,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-22 17:55:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:56,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:56,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:56,896 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:56,896 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:56,896 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:56,896 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:56,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,897 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,898 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,898 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:56,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:56,898 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,899 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:56,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:56,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:56,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:56,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:56,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:56,904 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,904 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:56,904 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:56,904 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:56,907 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:56,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1123b3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:56,909 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:56,909 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/31ff69c0fa014c619b49155da97a2364 2023-03-22 17:55:56,911 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:56,911 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,911 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:56,913 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:56,918 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:56,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:56,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:56,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:56,918 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:56,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:56,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:56,921 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/fbe18bee0a1746cb8f1eb37a9d5e0ad4 2023-03-22 17:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,923 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:56,927 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:56,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:56,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:56,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:56,927 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:56,927 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:56,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:56,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:56,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:56,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:56,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:56,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:56,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:56,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/09c320cd9f8e46d08d93aa5999f62657 2023-03-22 17:55:56,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/31ff69c0fa014c619b49155da97a2364 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/31ff69c0fa014c619b49155da97a2364 2023-03-22 17:55:56,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/31ff69c0fa014c619b49155da97a2364, entries=200, sequenceid=133, filesize=14.0 K 2023-03-22 17:55:56,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/fbe18bee0a1746cb8f1eb37a9d5e0ad4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fbe18bee0a1746cb8f1eb37a9d5e0ad4 2023-03-22 17:55:56,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fbe18bee0a1746cb8f1eb37a9d5e0ad4, entries=150, sequenceid=133, filesize=11.7 K 2023-03-22 17:55:56,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/09c320cd9f8e46d08d93aa5999f62657 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/09c320cd9f8e46d08d93aa5999f62657 2023-03-22 17:55:56,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/09c320cd9f8e46d08d93aa5999f62657, entries=150, sequenceid=133, filesize=11.7 K 2023-03-22 17:55:56,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=73.80 KB/75570 for 523fd4cd7256d4e27c9367c2949e4282 in 76ms, sequenceid=133, compaction requested=true 2023-03-22 17:55:56,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:56,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:56,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:56,966 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:56,966 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:55:56,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-22 17:55:56,968 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38304 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:56,969 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:55:56,969 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,969 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1b98d5acb4ed40d1a7795659aeb46c99, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/0cdc37ce18154e18999999e956cd2a1c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/31ff69c0fa014c619b49155da97a2364] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=37.4 K 2023-03-22 17:55:56,969 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:55:56,969 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:55:56,969 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1b98d5acb4ed40d1a7795659aeb46c99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679507755559 2023-03-22 17:55:56,969 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:56,970 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb9b0c4f73e445819a51a8618c4325ab, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca6c77931d5b45d2920917c762ea5779, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/320b138d0af3499288c641226014f2ec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/09c320cd9f8e46d08d93aa5999f62657] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=46.6 K 2023-03-22 17:55:56,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:56,971 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting bb9b0c4f73e445819a51a8618c4325ab, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679507755425 2023-03-22 17:55:56,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-22 17:55:56,971 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0cdc37ce18154e18999999e956cd2a1c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679507755711 2023-03-22 17:55:56,971 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:56,971 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,971 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:56,971 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,971 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:56,971 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:56,972 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ca6c77931d5b45d2920917c762ea5779, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679507755559 2023-03-22 17:55:56,972 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 31ff69c0fa014c619b49155da97a2364, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679507756355 2023-03-22 17:55:56,973 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 320b138d0af3499288c641226014f2ec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679507755711 2023-03-22 17:55:56,973 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 09c320cd9f8e46d08d93aa5999f62657, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679507756355 2023-03-22 17:55:56,985 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#402 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:56,989 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#403 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:56,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/8fd8dc3de1ee4aada330b48850cef896 2023-03-22 17:55:57,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/7d04e14997ac49969c02db82c4ce0daa 2023-03-22 17:55:57,013 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/271515a76d984653a7643ff150cb2ca0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/271515a76d984653a7643ff150cb2ca0 2023-03-22 17:55:57,016 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into 271515a76d984653a7643ff150cb2ca0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:57,017 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:57,017 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=13, startTime=1679507756966; duration=0sec 2023-03-22 17:55:57,017 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:57,017 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:57,019 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:57,020 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:55:57,020 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,020 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fd9ab0d76d334cde9866f6b19822d9dc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4aa12ec770b4b339aa2028ab4cdab07, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fbe18bee0a1746cb8f1eb37a9d5e0ad4] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=35.1 K 2023-03-22 17:55:57,021 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting fd9ab0d76d334cde9866f6b19822d9dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679507755559 2023-03-22 17:55:57,021 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c4aa12ec770b4b339aa2028ab4cdab07, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679507755711 2023-03-22 17:55:57,022 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting fbe18bee0a1746cb8f1eb37a9d5e0ad4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679507756355 2023-03-22 17:55:57,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/fdde32930d0a443ca358fd7e04548df8 2023-03-22 17:55:57,024 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/540b46a9293e4bf19ef77170e724462c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/540b46a9293e4bf19ef77170e724462c 2023-03-22 17:55:57,027 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/8fd8dc3de1ee4aada330b48850cef896 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fd8dc3de1ee4aada330b48850cef896 2023-03-22 17:55:57,028 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:57,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:57,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:57,030 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into 540b46a9293e4bf19ef77170e724462c(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:57,031 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:57,031 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=12, startTime=1679507756966; duration=0sec 2023-03-22 17:55:57,031 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:57,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507817022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507817029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,035 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:57,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:57,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:57,036 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:57,036 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,036 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:57,036 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:57,037 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,037 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,038 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,038 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fd8dc3de1ee4aada330b48850cef896, entries=200, sequenceid=148, filesize=14.1 K 2023-03-22 17:55:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:57,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/7d04e14997ac49969c02db82c4ce0daa as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7d04e14997ac49969c02db82c4ce0daa 2023-03-22 17:55:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:57,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7d04e14997ac49969c02db82c4ce0daa, entries=150, sequenceid=148, filesize=11.8 K 2023-03-22 17:55:57,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/fdde32930d0a443ca358fd7e04548df8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/fdde32930d0a443ca358fd7e04548df8 2023-03-22 17:55:57,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:57,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:57,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:57,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,044 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:57,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507817032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507817033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:57,048 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,048 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:57,049 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:57,049 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:57,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:57,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/fdde32930d0a443ca358fd7e04548df8, entries=150, sequenceid=148, filesize=11.8 K 2023-03-22 17:55:57,052 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 523fd4cd7256d4e27c9367c2949e4282 in 81ms, sequenceid=148, compaction requested=false 2023-03-22 17:55:57,052 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:57,053 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e503f88[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:57,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:57,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:57,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:57,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:57,060 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,060 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:57,061 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:57,070 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:57,070 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:57,070 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:57,070 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:57,070 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:57,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,072 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,072 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,072 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,074 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:57,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:57,076 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,076 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:57,076 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:57,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,076 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:57,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:57,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-22 17:55:57,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:57,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:57,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:57,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:57,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:57,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:57,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/c71f28e67a2f48bd94cb20fc2f9d5247 2023-03-22 17:55:57,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507817156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507817157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507817157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507817163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,176 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:57,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:57,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:57,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:57,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:57,179 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:57,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:57,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:57,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,182 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:57,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:57,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:57,184 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,184 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:57,184 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:57,184 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:57,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:57,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f75922d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:57,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:57,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,188 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:57,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:57,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:57,192 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:57,192 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:57,192 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:57,192 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:57,192 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:57,195 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,195 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,195 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,195 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,198 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,198 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,198 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,198 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:57,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:57,198 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,198 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,198 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:57,198 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:57,198 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,198 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:57,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507817268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507817268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507817268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507817270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,299 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:57,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:57,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:57,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:57,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:57,302 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:57,302 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,302 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:57,302 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:57,302 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,302 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,303 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,303 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:57,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:57,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:57,305 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,305 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:57,305 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:57,305 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:57,307 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,307 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,307 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:57,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:57,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121559b7[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:57,309 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,310 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:57,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:57,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:57,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,311 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:57,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:57,316 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:57,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:57,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:57,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:57,316 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:57,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,318 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,319 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:57,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:57,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,321 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,322 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,322 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:57,322 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:57,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,321 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:57,322 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,422 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:57,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:57,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:57,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:57,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:57,425 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:57,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:57,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:57,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,427 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:57,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:57,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:57,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:57,429 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,429 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:57,429 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:57,429 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:57,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:57,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:57,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717aecfd[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:57,432 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:57,432 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,436 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:57,436 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,437 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:57,436 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,438 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:57,440 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:57,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:57,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:57,441 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:57,441 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:57,442 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,442 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,442 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,443 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:57,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,447 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:57,446 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,447 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:57,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,455 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/f4262ea2347e4dc29f6360ff2ffea51d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f4262ea2347e4dc29f6360ff2ffea51d 2023-03-22 17:55:57,459 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into f4262ea2347e4dc29f6360ff2ffea51d(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:57,459 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:57,459 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=13, startTime=1679507756966; duration=0sec 2023-03-22 17:55:57,459 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:57,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507817472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507817472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507817475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507817476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,547 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:57,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:57,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:57,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:57,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:57,550 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:57,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:57,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:57,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,552 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:57,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:57,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:57,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:57,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:57,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:57,554 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,554 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:57,554 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:57,554 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:57,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:57,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:57,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:57,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/63a5de226aef46089330ffc901b905db 2023-03-22 17:55:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,557 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,557 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:57,557 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e793125[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:57,562 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:57,562 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,564 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:57,564 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:57,564 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:57,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,564 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:57,572 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/a603ee386afa4c6cba9878c2d01acc2c 2023-03-22 17:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:57,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:57,575 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:57,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:57,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:57,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:57,575 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:57,579 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,579 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,579 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:57,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:57,580 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/c71f28e67a2f48bd94cb20fc2f9d5247 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c71f28e67a2f48bd94cb20fc2f9d5247 2023-03-22 17:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:57,583 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c71f28e67a2f48bd94cb20fc2f9d5247, entries=250, sequenceid=169, filesize=16.4 K 2023-03-22 17:55:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:57,584 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/63a5de226aef46089330ffc901b905db as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/63a5de226aef46089330ffc901b905db 2023-03-22 17:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,587 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:57,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:57,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:57,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:57,587 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:57,587 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:57,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:57,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:57,588 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/63a5de226aef46089330ffc901b905db, entries=150, sequenceid=169, filesize=11.8 K 2023-03-22 17:55:57,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/a603ee386afa4c6cba9878c2d01acc2c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a603ee386afa4c6cba9878c2d01acc2c 2023-03-22 17:55:57,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a603ee386afa4c6cba9878c2d01acc2c, entries=150, sequenceid=169, filesize=11.8 K 2023-03-22 17:55:57,594 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 523fd4cd7256d4e27c9367c2949e4282 in 458ms, sequenceid=169, compaction requested=true 2023-03-22 17:55:57,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:57,595 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:57,595 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:57,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:57,596 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36316 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:57,596 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:55:57,596 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,596 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f4262ea2347e4dc29f6360ff2ffea51d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7d04e14997ac49969c02db82c4ce0daa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/63a5de226aef46089330ffc901b905db] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=35.5 K 2023-03-22 17:55:57,596 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43486 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:57,597 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:55:57,597 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f4262ea2347e4dc29f6360ff2ffea51d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679507756355 2023-03-22 17:55:57,597 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,597 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/271515a76d984653a7643ff150cb2ca0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fd8dc3de1ee4aada330b48850cef896, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c71f28e67a2f48bd94cb20fc2f9d5247] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=42.5 K 2023-03-22 17:55:57,598 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7d04e14997ac49969c02db82c4ce0daa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679507756956 2023-03-22 17:55:57,598 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 271515a76d984653a7643ff150cb2ca0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679507756355 2023-03-22 17:55:57,598 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 63a5de226aef46089330ffc901b905db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507757031 2023-03-22 17:55:57,598 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 8fd8dc3de1ee4aada330b48850cef896, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679507756948 2023-03-22 17:55:57,600 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c71f28e67a2f48bd94cb20fc2f9d5247, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507757017 2023-03-22 17:55:57,605 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#410 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:57,611 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#411 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:57,687 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:57,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:57,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:57,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:57,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:57,700 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:57,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:57,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:57,700 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,701 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,702 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:57,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:57,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:57,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:57,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:57,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:57,703 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:57,704 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:57,704 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:57,704 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:57,704 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:57,704 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-22 17:55:57,704 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:57,704 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:57,704 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:57,704 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:57,704 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:57,704 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:57,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:57,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507817801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507817802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507817803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507817804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507817805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507817912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507817912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507817912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507817912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:57,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:57,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507817913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,025 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/9fe2410beea54d86b6c812efc5b584e6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9fe2410beea54d86b6c812efc5b584e6 2023-03-22 17:55:58,028 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/f6f1ad4de271469ebd35414282d0f3cd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f6f1ad4de271469ebd35414282d0f3cd 2023-03-22 17:55:58,032 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into 9fe2410beea54d86b6c812efc5b584e6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:58,032 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:58,032 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=13, startTime=1679507757595; duration=0sec 2023-03-22 17:55:58,033 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:55:58,033 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:55:58,034 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36282 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:55:58,034 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:55:58,034 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:58,034 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/540b46a9293e4bf19ef77170e724462c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/fdde32930d0a443ca358fd7e04548df8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a603ee386afa4c6cba9878c2d01acc2c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=35.4 K 2023-03-22 17:55:58,034 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into f6f1ad4de271469ebd35414282d0f3cd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:58,034 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:58,034 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=13, startTime=1679507757595; duration=0sec 2023-03-22 17:55:58,034 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:58,034 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 540b46a9293e4bf19ef77170e724462c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679507756355 2023-03-22 17:55:58,035 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting fdde32930d0a443ca358fd7e04548df8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679507756956 2023-03-22 17:55:58,035 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a603ee386afa4c6cba9878c2d01acc2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507757031 2023-03-22 17:55:58,040 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#413 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:55:58,048 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/e443418e7b01422c848983ab896e10fa as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e443418e7b01422c848983ab896e10fa 2023-03-22 17:55:58,051 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into e443418e7b01422c848983ab896e10fa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:55:58,051 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:58,051 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=13, startTime=1679507757595; duration=0sec 2023-03-22 17:55:58,052 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:55:58,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507818116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507818117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507818117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507818118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507818118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,130 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/552b544636cb44518cccc2c9fca9ff48 2023-03-22 17:55:58,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507818418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507818422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507818422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507818423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507818423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,541 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/033f384611f645749f56cc0e097486ec 2023-03-22 17:55:58,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507818925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507818926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507818927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507818928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:55:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507818928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,949 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/d7c78d749c6843b6874c6e0ec477ee6d 2023-03-22 17:55:58,952 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/552b544636cb44518cccc2c9fca9ff48 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/552b544636cb44518cccc2c9fca9ff48 2023-03-22 17:55:58,955 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/552b544636cb44518cccc2c9fca9ff48, entries=150, sequenceid=183, filesize=11.8 K 2023-03-22 17:55:58,956 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/033f384611f645749f56cc0e097486ec as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/033f384611f645749f56cc0e097486ec 2023-03-22 17:55:58,959 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/033f384611f645749f56cc0e097486ec, entries=150, sequenceid=183, filesize=11.8 K 2023-03-22 17:55:58,960 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/d7c78d749c6843b6874c6e0ec477ee6d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d7c78d749c6843b6874c6e0ec477ee6d 2023-03-22 17:55:58,962 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d7c78d749c6843b6874c6e0ec477ee6d, entries=150, sequenceid=183, filesize=11.8 K 2023-03-22 17:55:58,963 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 523fd4cd7256d4e27c9367c2949e4282 in 1259ms, sequenceid=183, compaction requested=false 2023-03-22 17:55:58,963 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:58,963 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:58,965 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:58,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:58,965 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d3e343d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:58,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:58,967 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:58,968 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:58,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:58,968 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:58,968 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:58,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:58,968 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:58,970 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:58,970 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:58,970 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,970 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:58,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:58,972 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:58,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:58,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:58,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:58,973 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:58,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:58,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:58,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:58,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:58,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:58,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:58,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:58,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:58,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:58,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:58,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:58,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:58,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:58,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:58,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:58,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:58,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:58,979 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:58,979 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:58,979 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:58,979 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:58,979 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:58,979 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,079 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:59,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:59,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:59,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:59,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:59,082 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:59,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:59,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:59,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:59,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,084 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:59,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:59,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:59,084 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,085 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:59,085 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:59,086 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-22 17:55:59,086 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,086 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:59,497 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/b4ff3eea01db482e8086092dd9d52411 2023-03-22 17:55:59,507 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/edf921d2c0724d999982ab8a6f4763ae 2023-03-22 17:55:59,517 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/5e52e6f3806747698cddb5183d35ec55 2023-03-22 17:55:59,520 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/b4ff3eea01db482e8086092dd9d52411 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b4ff3eea01db482e8086092dd9d52411 2023-03-22 17:55:59,522 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b4ff3eea01db482e8086092dd9d52411, entries=150, sequenceid=205, filesize=11.8 K 2023-03-22 17:55:59,523 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/edf921d2c0724d999982ab8a6f4763ae as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/edf921d2c0724d999982ab8a6f4763ae 2023-03-22 17:55:59,526 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/edf921d2c0724d999982ab8a6f4763ae, entries=150, sequenceid=205, filesize=11.8 K 2023-03-22 17:55:59,527 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/5e52e6f3806747698cddb5183d35ec55 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5e52e6f3806747698cddb5183d35ec55 2023-03-22 17:55:59,529 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5e52e6f3806747698cddb5183d35ec55, entries=150, sequenceid=205, filesize=11.8 K 2023-03-22 17:55:59,530 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 523fd4cd7256d4e27c9367c2949e4282 in 444ms, sequenceid=205, compaction requested=true 2023-03-22 17:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:59,530 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:59,538 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:59,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@455e02cc[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:59,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:59,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,541 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:59,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:59,547 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:59,549 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:59,549 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:59,549 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:59,549 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:59,549 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:59,552 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,552 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,552 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:59,553 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,554 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,556 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:59,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:59,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:59,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:59,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:59,556 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:59,556 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:59,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:59,656 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:59,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:59,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:59,659 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:59,659 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:59,659 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:59,659 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:59,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:59,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:59,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:59,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:59,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:59,663 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:59,663 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:59,663 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:59,663 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:59,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:59,665 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:59,667 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e5039e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:59,667 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,669 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,669 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:59,669 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:59,669 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:59,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:59,670 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:59,673 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:59,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:59,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:59,673 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:59,673 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:59,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:59,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,675 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,676 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,678 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:59,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:59,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:59,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:59,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:59,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:59,679 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:59,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,679 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:59,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:59,779 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:59,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:59,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:59,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:59,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:59,782 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:59,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:59,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:59,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:59,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,785 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:59,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:59,786 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,786 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:59,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:59,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:59,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:59,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:55:59,787 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:55:59,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:55:59,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,789 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,789 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,789 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:55:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:55:59,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7402e6de[Count = 0] remaining members to acquire global barrier 2023-03-22 17:55:59,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:55:59,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:55:59,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:55:59,792 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:55:59,792 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,792 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:55:59,793 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:55:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:55:59,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:55:59,796 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:55:59,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:55:59,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:55:59,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:55:59,796 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:55:59,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:55:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:55:59,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:59,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:55:59,799 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:55:59,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,800 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:55:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:55:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,802 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,802 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,802 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:59,802 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,802 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:55:59,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:59,802 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:55:59,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:55:59,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:55:59,802 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:55:59,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,803 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:55:59,803 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:55:59,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:55:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:55:59,903 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:55:59,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:55:59,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:55:59,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:55:59,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:55:59,932 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:55:59,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:59,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:55:59,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:55:59,933 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,933 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,934 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,934 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:55:59,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:55:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:55:59,934 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:55:59,934 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:55:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:55:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:55:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:55:59,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:55:59,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:55:59,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:55:59,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:55:59,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:55:59,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:55:59,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:55:59,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:55:59,936 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:55:59,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:55:59,937 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-22 17:55:59,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:55:59,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:59,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:55:59,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:59,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:55:59,937 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:55:59,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:55:59,947 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:55:59,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507759940, max timestamp=1679507759963Num uniques 859; 2023-03-22 17:55:59,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:59,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507759940, max timestamp=1679507759963Num uniques 859; 2023-03-22 17:55:59,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:59,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507759940, max timestamp=1679507759963Num uniques 859; 2023-03-22 17:55:59,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:55:59,966 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/138c50bf60354ddf83d1962b4dea51d0 2023-03-22 17:55:59,981 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/f13c0d350a174c20bac24a526d179960 2023-03-22 17:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679507759980, max timestamp=1679507759995Num uniques 1068; 2023-03-22 17:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-03-22 17:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679507759980, max timestamp=1679507759995Num uniques 1068; 2023-03-22 17:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-03-22 17:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679507759980, max timestamp=1679507759995Num uniques 1068; 2023-03-22 17:55:59,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-03-22 17:56:00,002 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/d8e3d770c26c474882e549d8937bd57d 2023-03-22 17:56:00,007 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/138c50bf60354ddf83d1962b4dea51d0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/138c50bf60354ddf83d1962b4dea51d0 2023-03-22 17:56:00,012 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/138c50bf60354ddf83d1962b4dea51d0, entries=50, sequenceid=211, filesize=7.1 K 2023-03-22 17:56:00,013 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/f13c0d350a174c20bac24a526d179960 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f13c0d350a174c20bac24a526d179960 2023-03-22 17:56:00,020 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f13c0d350a174c20bac24a526d179960, entries=50, sequenceid=211, filesize=7.1 K 2023-03-22 17:56:00,021 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/d8e3d770c26c474882e549d8937bd57d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d8e3d770c26c474882e549d8937bd57d 2023-03-22 17:56:00,026 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d8e3d770c26c474882e549d8937bd57d, entries=50, sequenceid=211, filesize=7.1 K 2023-03-22 17:56:00,026 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 89ms, sequenceid=211, compaction requested=true 2023-03-22 17:56:00,026 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:00,027 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:00,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:00,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=140.89 KB heapSize=385.31 KB 2023-03-22 17:56:00,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:00,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:00,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:00,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:00,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:00,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:00,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7811abbf[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:00,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:00,031 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,032 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,032 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:00,032 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:00,033 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:00,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:00,035 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:00,035 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,035 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:00,035 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:00,038 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:00,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:00,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:00,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:00,039 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:00,041 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,041 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:00,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,041 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:00,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,043 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,048 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:00,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:00,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:00,049 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:00,049 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:00,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/9145089327f449d0be732c8ab5d9ccaa 2023-03-22 17:56:00,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:00,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:00,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:00,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:00,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507820041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507820054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507820043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507820054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/bf3797b34c6f486fb1c0130f5690f9a9 2023-03-22 17:56:00,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507820058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/f477e5d2c82c4ee185ab054fe73cd00c 2023-03-22 17:56:00,084 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/9145089327f449d0be732c8ab5d9ccaa as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9145089327f449d0be732c8ab5d9ccaa 2023-03-22 17:56:00,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9145089327f449d0be732c8ab5d9ccaa, entries=250, sequenceid=262, filesize=16.6 K 2023-03-22 17:56:00,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/bf3797b34c6f486fb1c0130f5690f9a9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/bf3797b34c6f486fb1c0130f5690f9a9 2023-03-22 17:56:00,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/bf3797b34c6f486fb1c0130f5690f9a9, entries=150, sequenceid=262, filesize=11.9 K 2023-03-22 17:56:00,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/f477e5d2c82c4ee185ab054fe73cd00c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f477e5d2c82c4ee185ab054fe73cd00c 2023-03-22 17:56:00,103 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f477e5d2c82c4ee185ab054fe73cd00c, entries=150, sequenceid=262, filesize=11.9 K 2023-03-22 17:56:00,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~385.27 KB/394512, currentSize=46.96 KB/48090 for 523fd4cd7256d4e27c9367c2949e4282 in 77ms, sequenceid=262, compaction requested=true 2023-03-22 17:56:00,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:00,104 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:00,104 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:00,104 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:00,104 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:00,104 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:00,105 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60784 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:00,105 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55954 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:00,105 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:56:00,105 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:56:00,105 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:00,106 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:00,106 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f6f1ad4de271469ebd35414282d0f3cd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/552b544636cb44518cccc2c9fca9ff48, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b4ff3eea01db482e8086092dd9d52411, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/138c50bf60354ddf83d1962b4dea51d0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9145089327f449d0be732c8ab5d9ccaa] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=59.4 K 2023-03-22 17:56:00,106 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9fe2410beea54d86b6c812efc5b584e6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/033f384611f645749f56cc0e097486ec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/edf921d2c0724d999982ab8a6f4763ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f13c0d350a174c20bac24a526d179960, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/bf3797b34c6f486fb1c0130f5690f9a9] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=54.6 K 2023-03-22 17:56:00,106 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f6f1ad4de271469ebd35414282d0f3cd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507757031 2023-03-22 17:56:00,106 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9fe2410beea54d86b6c812efc5b584e6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507757031 2023-03-22 17:56:00,106 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 552b544636cb44518cccc2c9fca9ff48, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679507757155 2023-03-22 17:56:00,106 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 033f384611f645749f56cc0e097486ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679507757155 2023-03-22 17:56:00,106 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b4ff3eea01db482e8086092dd9d52411, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507757796 2023-03-22 17:56:00,107 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting edf921d2c0724d999982ab8a6f4763ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507757796 2023-03-22 17:56:00,107 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 138c50bf60354ddf83d1962b4dea51d0, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1679507759933 2023-03-22 17:56:00,107 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f13c0d350a174c20bac24a526d179960, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1679507759933 2023-03-22 17:56:00,107 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 9145089327f449d0be732c8ab5d9ccaa, keycount=250, bloomtype=ROW, size=16.6 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679507760020 2023-03-22 17:56:00,108 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting bf3797b34c6f486fb1c0130f5690f9a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679507760023 2023-03-22 17:56:00,122 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:00,128 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:00,140 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/1e6b7f5e51484f3990ca72124568ecd9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1e6b7f5e51484f3990ca72124568ecd9 2023-03-22 17:56:00,146 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into 1e6b7f5e51484f3990ca72124568ecd9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:00,146 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:00,147 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=11, startTime=1679507760104; duration=0sec 2023-03-22 17:56:00,147 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:00,147 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:00,149 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55920 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:00,149 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:56:00,149 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:00,149 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e443418e7b01422c848983ab896e10fa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d7c78d749c6843b6874c6e0ec477ee6d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5e52e6f3806747698cddb5183d35ec55, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d8e3d770c26c474882e549d8937bd57d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f477e5d2c82c4ee185ab054fe73cd00c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=54.6 K 2023-03-22 17:56:00,149 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:00,150 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e443418e7b01422c848983ab896e10fa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679507757031 2023-03-22 17:56:00,150 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d7c78d749c6843b6874c6e0ec477ee6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679507757155 2023-03-22 17:56:00,150 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5e52e6f3806747698cddb5183d35ec55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1679507757796 2023-03-22 17:56:00,151 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d8e3d770c26c474882e549d8937bd57d, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1679507759933 2023-03-22 17:56:00,151 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f477e5d2c82c4ee185ab054fe73cd00c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679507760023 2023-03-22 17:56:00,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:00,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:00,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:00,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:00,154 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:00,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:00,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:00,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:00,155 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,155 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,156 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:00,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:00,157 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,157 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:00,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:00,160 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:00,160 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:00,160 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:00,160 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:00,160 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:00,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:00,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:00,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:00,161 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:00,170 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#428 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:00,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:00,171 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:00,174 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/083f4c71369d426198c6d2cd580fe773 2023-03-22 17:56:00,180 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/e9a5ebcadd7b4c92b1bd8a77ab15000a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e9a5ebcadd7b4c92b1bd8a77ab15000a 2023-03-22 17:56:00,185 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into e9a5ebcadd7b4c92b1bd8a77ab15000a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:00,185 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:00,185 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=11, startTime=1679507760104; duration=0sec 2023-03-22 17:56:00,185 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:00,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507760186, max timestamp=1679507760199Num uniques 859; 2023-03-22 17:56:00,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:00,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507760186, max timestamp=1679507760199Num uniques 859; 2023-03-22 17:56:00,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507760186, max timestamp=1679507760199Num uniques 859; 2023-03-22 17:56:00,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:00,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:00,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507820226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507820227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507820227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507820228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507820235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507820336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507820336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507820336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507820336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507820345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,542 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/1c8e67a3ad964609a7ef6301fec74641 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1c8e67a3ad964609a7ef6301fec74641 2023-03-22 17:56:00,547 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into 1c8e67a3ad964609a7ef6301fec74641(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:00,547 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:00,547 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=11, startTime=1679507760104; duration=0sec 2023-03-22 17:56:00,547 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:00,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507820544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507820544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507820545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507820545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507820555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,585 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/20afb051e550486fa29b6f542e91473f 2023-03-22 17:56:00,597 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/1eb4b5c4d1ea42d88fb4afcdbabc04f4 2023-03-22 17:56:00,602 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/083f4c71369d426198c6d2cd580fe773 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/083f4c71369d426198c6d2cd580fe773 2023-03-22 17:56:00,606 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/083f4c71369d426198c6d2cd580fe773, entries=150, sequenceid=273, filesize=11.9 K 2023-03-22 17:56:00,607 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/20afb051e550486fa29b6f542e91473f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/20afb051e550486fa29b6f542e91473f 2023-03-22 17:56:00,616 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/20afb051e550486fa29b6f542e91473f, entries=150, sequenceid=273, filesize=11.9 K 2023-03-22 17:56:00,619 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/1eb4b5c4d1ea42d88fb4afcdbabc04f4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/1eb4b5c4d1ea42d88fb4afcdbabc04f4 2023-03-22 17:56:00,626 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/1eb4b5c4d1ea42d88fb4afcdbabc04f4, entries=150, sequenceid=273, filesize=11.9 K 2023-03-22 17:56:00,628 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 467ms, sequenceid=273, compaction requested=false 2023-03-22 17:56:00,628 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:00,628 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d6c9e13[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:00,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:00,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,637 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:00,637 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,637 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:00,638 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:00,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:00,641 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:00,641 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:00,641 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:00,641 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:00,641 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:00,642 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,642 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:00,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,643 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:00,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,643 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,647 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:00,647 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:00,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:00,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:00,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:00,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:00,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:00,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:00,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:00,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,648 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:00,648 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:00,748 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:00,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:00,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:00,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:00,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:00,751 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:00,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:00,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:00,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:00,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:00,754 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:00,754 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,755 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:00,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:00,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:00,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:00,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:00,756 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:00,756 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:00,756 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-22 17:56:00,756 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:00,756 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:00,756 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:00,756 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:00,756 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:00,756 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:00,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:00,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:00,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507820861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507820862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507820864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507820866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507820867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507820973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507820974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507820974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:00,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507820974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,170 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/123ecd74bf4345a08afd4d2494013493 2023-03-22 17:56:01,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54980 deadline: 1679507821180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507821183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507821183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507821184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507821375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507821488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507821492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507821492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:01,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55012 deadline: 1679507821494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,594 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/6bd18fb515754a39801403c9955d2ef6 2023-03-22 17:56:01,611 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/c71a52dbe6b84a2f8055ae212130b646 2023-03-22 17:56:01,616 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/123ecd74bf4345a08afd4d2494013493 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/123ecd74bf4345a08afd4d2494013493 2023-03-22 17:56:01,619 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/123ecd74bf4345a08afd4d2494013493, entries=150, sequenceid=311, filesize=11.9 K 2023-03-22 17:56:01,620 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/6bd18fb515754a39801403c9955d2ef6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6bd18fb515754a39801403c9955d2ef6 2023-03-22 17:56:01,623 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6bd18fb515754a39801403c9955d2ef6, entries=150, sequenceid=311, filesize=11.9 K 2023-03-22 17:56:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/c71a52dbe6b84a2f8055ae212130b646 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c71a52dbe6b84a2f8055ae212130b646 2023-03-22 17:56:01,645 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c71a52dbe6b84a2f8055ae212130b646, entries=150, sequenceid=311, filesize=11.9 K 2023-03-22 17:56:01,647 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 523fd4cd7256d4e27c9367c2949e4282 in 891ms, sequenceid=311, compaction requested=true 2023-03-22 17:56:01,647 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:01,647 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:01,647 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:01,647 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:01,647 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,647 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:01,648 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:01,649 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:01,649 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:01,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:01,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6de8dd3[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:01,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:01,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:01,655 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:01,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:01,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:01,656 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,656 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:01,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:01,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:01,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:01,660 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:01,664 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:01,664 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:01,664 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:01,664 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:01,664 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:01,667 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,667 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:01,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,667 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:01,668 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:01,674 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:01,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:01,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:01,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:01,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:01,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:01,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:01,675 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:01,675 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:01,675 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:01,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:01,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:01,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,776 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:01,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:01,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:01,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:01,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:01,791 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:01,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:01,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:01,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:01,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,791 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:01,793 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:01,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:01,793 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:01,793 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:01,793 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:01,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:01,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:01,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:01,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:01,803 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:01,803 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:01,803 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:01,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:01,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:01,820 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:01,820 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:01,820 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:01,820 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:01,820 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:01,821 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:01,821 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:01,821 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:01,821 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:01,872 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/a6cb260f3ff8465f919aa7c846a2fe3b 2023-03-22 17:56:01,942 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/5b3e7ed920214dc095b4670ba43c0fe2 2023-03-22 17:56:02,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:02,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:02,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507762043, max timestamp=1679507762063Num uniques 859; 2023-03-22 17:56:02,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507762043, max timestamp=1679507762063Num uniques 859; 2023-03-22 17:56:02,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:56:02,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507762043, max timestamp=1679507762063Num uniques 859; 2023-03-22 17:56:02,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:56:02,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:56:02,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507822101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507822116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507822125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54970 deadline: 1679507822125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507822226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507822227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507822239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507822239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507822384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,398 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/651aa0e5fb50496c8b30295eeefaaefb 2023-03-22 17:56:02,402 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/a6cb260f3ff8465f919aa7c846a2fe3b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a6cb260f3ff8465f919aa7c846a2fe3b 2023-03-22 17:56:02,406 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a6cb260f3ff8465f919aa7c846a2fe3b, entries=100, sequenceid=321, filesize=9.5 K 2023-03-22 17:56:02,411 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/5b3e7ed920214dc095b4670ba43c0fe2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5b3e7ed920214dc095b4670ba43c0fe2 2023-03-22 17:56:02,415 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5b3e7ed920214dc095b4670ba43c0fe2, entries=100, sequenceid=321, filesize=9.5 K 2023-03-22 17:56:02,416 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/651aa0e5fb50496c8b30295eeefaaefb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/651aa0e5fb50496c8b30295eeefaaefb 2023-03-22 17:56:02,419 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/651aa0e5fb50496c8b30295eeefaaefb, entries=100, sequenceid=321, filesize=9.5 K 2023-03-22 17:56:02,420 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 600ms, sequenceid=321, compaction requested=true 2023-03-22 17:56:02,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:02,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,425 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6589bfc0[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:02,428 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:02,428 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,429 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:02,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:02,430 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,430 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:02,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:02,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:02,435 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:02,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:02,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:02,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:02,435 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:02,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:02,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,437 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,438 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-22 17:56:02,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:02,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:02,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:02,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:02,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:02,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:02,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,440 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:02,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:02,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:02,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:02,446 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:02,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:02,446 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:02,447 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:02,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,447 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:02,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:02,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507822448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507822451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507822452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507822456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/347496a40f434e76bde7238b13861927 2023-03-22 17:56:02,502 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/de3238e6a6584cf58b45cd7a72c3b9aa 2023-03-22 17:56:02,547 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:02,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:02,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507822557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:02,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:02,572 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:02,572 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:02,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,572 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:02,572 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:02,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507822564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,574 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,574 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,575 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:02,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:02,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,575 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:02,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:02,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:02,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:02,594 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:02,605 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:02,605 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:02,605 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:02,605 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:02,607 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,609 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b0bc9d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:02,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:02,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:02,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:02,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:02,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:02,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:02,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:02,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:02,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:02,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:02,621 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:02,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:02,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:02,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:02,621 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:02,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:02,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:02,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:02,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:02,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,630 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:02,630 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:02,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:02,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:02,730 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:02,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:02,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:02,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:02,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:02,736 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:02,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:02,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:02,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:02,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,738 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:02,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:02,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:02,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:02,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:02,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:02,757 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:02,758 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:02,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507822764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507822764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,779 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:02,779 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:02,779 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:02,779 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:02,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:02,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:02,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:02,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:02,781 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,781 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3985349e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:02,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:02,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,784 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,785 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:02,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:02,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:02,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:02,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507822773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:02,786 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507822773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:02,787 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:02,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:02,789 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:02,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:02,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:02,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:02,789 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:02,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:02,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,793 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,794 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:02,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,797 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:02,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:02,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:02,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:02,798 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:02,798 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:02,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:02,898 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:02,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:02,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:02,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:02,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:02,903 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:02,903 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:02,903 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:02,903 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:02,904 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,904 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,905 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:02,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:02,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:02,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:02,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:02,915 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:02,915 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:02,915 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:02,915 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:02,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:02,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:02,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:02,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:02,918 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,918 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,920 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,920 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:02,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:02,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b479668[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:02,922 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:02,922 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,924 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,925 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:02,925 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:02,925 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:02,926 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:02,926 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:02,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:02,927 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:02,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:02,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:02,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:02,927 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:02,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:02,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:02,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,936 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,939 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:02,939 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:02,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:02,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:02,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:02,940 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:02,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:02,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:02,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:02,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:02,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:02,940 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:02,940 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:02,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/bdb4299702d34d95978fa1a1059bc3ae 2023-03-22 17:56:03,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/347496a40f434e76bde7238b13861927 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/347496a40f434e76bde7238b13861927 2023-03-22 17:56:03,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/347496a40f434e76bde7238b13861927, entries=200, sequenceid=357, filesize=14.3 K 2023-03-22 17:56:03,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/de3238e6a6584cf58b45cd7a72c3b9aa as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/de3238e6a6584cf58b45cd7a72c3b9aa 2023-03-22 17:56:03,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/de3238e6a6584cf58b45cd7a72c3b9aa, entries=150, sequenceid=357, filesize=11.9 K 2023-03-22 17:56:03,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/bdb4299702d34d95978fa1a1059bc3ae as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bdb4299702d34d95978fa1a1059bc3ae 2023-03-22 17:56:03,040 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:03,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:03,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:03,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bdb4299702d34d95978fa1a1059bc3ae, entries=150, sequenceid=357, filesize=11.9 K 2023-03-22 17:56:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:03,048 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:03,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:03,048 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 523fd4cd7256d4e27c9367c2949e4282 in 610ms, sequenceid=357, compaction requested=true 2023-03-22 17:56:03,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:03,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:03,048 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:03,048 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:03,048 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:03,048 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:03,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:03,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:03,048 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:03,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:03,051 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61506 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:03,051 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:56:03,051 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:03,051 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1e6b7f5e51484f3990ca72124568ecd9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/083f4c71369d426198c6d2cd580fe773, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/123ecd74bf4345a08afd4d2494013493, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a6cb260f3ff8465f919aa7c846a2fe3b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/347496a40f434e76bde7238b13861927] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=60.1 K 2023-03-22 17:56:03,052 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59066 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:03,052 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:56:03,052 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:03,052 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1c8e67a3ad964609a7ef6301fec74641, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/20afb051e550486fa29b6f542e91473f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6bd18fb515754a39801403c9955d2ef6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5b3e7ed920214dc095b4670ba43c0fe2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/de3238e6a6584cf58b45cd7a72c3b9aa] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=57.7 K 2023-03-22 17:56:03,052 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1e6b7f5e51484f3990ca72124568ecd9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679507760023 2023-03-22 17:56:03,052 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 083f4c71369d426198c6d2cd580fe773, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1679507760036 2023-03-22 17:56:03,053 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1c8e67a3ad964609a7ef6301fec74641, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679507760023 2023-03-22 17:56:03,053 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 123ecd74bf4345a08afd4d2494013493, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1679507760226 2023-03-22 17:56:03,053 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a6cb260f3ff8465f919aa7c846a2fe3b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679507760865 2023-03-22 17:56:03,053 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 20afb051e550486fa29b6f542e91473f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1679507760036 2023-03-22 17:56:03,053 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 347496a40f434e76bde7238b13861927, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679507762111 2023-03-22 17:56:03,055 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6bd18fb515754a39801403c9955d2ef6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1679507760226 2023-03-22 17:56:03,056 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 5b3e7ed920214dc095b4670ba43c0fe2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679507760865 2023-03-22 17:56:03,057 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting de3238e6a6584cf58b45cd7a72c3b9aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679507762111 2023-03-22 17:56:03,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:03,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:03,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:03,059 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,059 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:03,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:03,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:03,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:03,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:03,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:03,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:03,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:03,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:03,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:03,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:03,076 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:03,076 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:03,076 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:03,076 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:03,076 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-22 17:56:03,076 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:03,076 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:03,077 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:03,077 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:03,077 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:03,077 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:03,084 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:03,085 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:03,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:03,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:03,129 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/e9bd3a9074dc482d986b19a70fd5d758 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/e9bd3a9074dc482d986b19a70fd5d758 2023-03-22 17:56:03,133 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into e9bd3a9074dc482d986b19a70fd5d758(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:03,133 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:03,133 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=11, startTime=1679507763048; duration=0sec 2023-03-22 17:56:03,133 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:03,133 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:03,135 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59032 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:03,135 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:56:03,135 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:03,135 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e9a5ebcadd7b4c92b1bd8a77ab15000a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/1eb4b5c4d1ea42d88fb4afcdbabc04f4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c71a52dbe6b84a2f8055ae212130b646, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/651aa0e5fb50496c8b30295eeefaaefb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bdb4299702d34d95978fa1a1059bc3ae] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=57.6 K 2023-03-22 17:56:03,135 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e9a5ebcadd7b4c92b1bd8a77ab15000a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679507760023 2023-03-22 17:56:03,135 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1eb4b5c4d1ea42d88fb4afcdbabc04f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1679507760036 2023-03-22 17:56:03,136 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c71a52dbe6b84a2f8055ae212130b646, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1679507760226 2023-03-22 17:56:03,136 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 651aa0e5fb50496c8b30295eeefaaefb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679507760865 2023-03-22 17:56:03,136 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting bdb4299702d34d95978fa1a1059bc3ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679507762111 2023-03-22 17:56:03,149 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507763156, max timestamp=1679507763187Num uniques 859; 2023-03-22 17:56:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:03,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507763156, max timestamp=1679507763187Num uniques 859; 2023-03-22 17:56:03,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:03,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507763156, max timestamp=1679507763187Num uniques 859; 2023-03-22 17:56:03,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:03,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507763242, max timestamp=1679507763272Num uniques 1018; 2023-03-22 17:56:03,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507763242, max timestamp=1679507763272Num uniques 1018; 2023-03-22 17:56:03,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-22 17:56:03,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-22 17:56:03,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507763242, max timestamp=1679507763272Num uniques 1018; 2023-03-22 17:56:03,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-22 17:56:03,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507823314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507823323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507823324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507823326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507823430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507823440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507823440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507823440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,538 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/7f56a5ded9394cac965dc49fb9c0076c 2023-03-22 17:56:03,551 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/5e8c79d4924d4dbcb08474ac3c53a070 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5e8c79d4924d4dbcb08474ac3c53a070 2023-03-22 17:56:03,565 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into 5e8c79d4924d4dbcb08474ac3c53a070(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:03,565 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:03,565 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=11, startTime=1679507763048; duration=0sec 2023-03-22 17:56:03,565 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:03,565 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/4f83eb768c8d45b99728534dc94cdbd0 2023-03-22 17:56:03,569 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/9a5615ea28834320a63b10b1d27ca4a3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/9a5615ea28834320a63b10b1d27ca4a3 2023-03-22 17:56:03,573 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into 9a5615ea28834320a63b10b1d27ca4a3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:03,573 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:03,573 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=11, startTime=1679507763049; duration=0sec 2023-03-22 17:56:03,573 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:03,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507823640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507823649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507823650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507823650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507823947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507823952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507823954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:03,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507823954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:03,984 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/c66468b224c84e6185599bcb57fcc0b1 2023-03-22 17:56:03,988 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/7f56a5ded9394cac965dc49fb9c0076c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/7f56a5ded9394cac965dc49fb9c0076c 2023-03-22 17:56:03,991 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/7f56a5ded9394cac965dc49fb9c0076c, entries=150, sequenceid=366, filesize=11.9 K 2023-03-22 17:56:03,992 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/4f83eb768c8d45b99728534dc94cdbd0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/4f83eb768c8d45b99728534dc94cdbd0 2023-03-22 17:56:03,996 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/4f83eb768c8d45b99728534dc94cdbd0, entries=150, sequenceid=366, filesize=11.9 K 2023-03-22 17:56:03,996 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/c66468b224c84e6185599bcb57fcc0b1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c66468b224c84e6185599bcb57fcc0b1 2023-03-22 17:56:04,000 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c66468b224c84e6185599bcb57fcc0b1, entries=150, sequenceid=366, filesize=11.9 K 2023-03-22 17:56:04,000 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 523fd4cd7256d4e27c9367c2949e4282 in 924ms, sequenceid=366, compaction requested=false 2023-03-22 17:56:04,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:04,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:04,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:04,006 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:04,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6398577f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:04,007 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:04,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,009 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,009 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,009 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:04,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:04,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:04,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:04,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:04,015 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:04,015 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:04,015 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:04,015 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:04,015 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:04,020 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,020 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,020 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,023 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,026 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,026 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,026 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:04,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:04,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:04,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,027 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:04,027 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:04,127 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:04,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:04,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:04,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:04,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:04,131 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:04,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:04,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:04,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,132 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:04,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:04,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:04,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:04,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:04,136 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:04,136 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,136 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:04,140 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:04,140 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:04,140 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,140 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:04,140 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-03-22 17:56:04,140 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:04,140 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:04,141 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:04,141 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:04,141 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:04,141 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:04,160 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/85c94bff4e8b489482f394da09b6493b 2023-03-22 17:56:04,189 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/0f22b662b6974d5c98df9aa7feff7f3a 2023-03-22 17:56:04,232 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/a6ab4778fca3462eae88193b8ac79083 2023-03-22 17:56:04,236 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/85c94bff4e8b489482f394da09b6493b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/85c94bff4e8b489482f394da09b6493b 2023-03-22 17:56:04,240 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/85c94bff4e8b489482f394da09b6493b, entries=150, sequenceid=423, filesize=11.9 K 2023-03-22 17:56:04,241 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/0f22b662b6974d5c98df9aa7feff7f3a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0f22b662b6974d5c98df9aa7feff7f3a 2023-03-22 17:56:04,252 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0f22b662b6974d5c98df9aa7feff7f3a, entries=150, sequenceid=423, filesize=11.9 K 2023-03-22 17:56:04,253 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/a6ab4778fca3462eae88193b8ac79083 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a6ab4778fca3462eae88193b8ac79083 2023-03-22 17:56:04,266 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a6ab4778fca3462eae88193b8ac79083, entries=150, sequenceid=423, filesize=11.9 K 2023-03-22 17:56:04,267 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=0 B/0 for 523fd4cd7256d4e27c9367c2949e4282 in 127ms, sequenceid=423, compaction requested=true 2023-03-22 17:56:04,267 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:04,267 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:04,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,273 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:04,273 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:04,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76d94bc5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:04,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:04,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,280 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,281 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:04,281 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:04,281 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:04,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:04,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:04,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:04,283 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:04,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:04,289 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:04,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:04,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:04,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:04,289 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:04,291 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,291 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,291 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,293 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:04,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:04,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,298 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,298 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,298 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,298 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:04,298 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:04,398 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:04,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:04,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:04,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:04,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:04,403 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:04,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:04,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:04,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,405 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:04,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:04,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:04,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:04,416 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:04,416 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:04,416 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,416 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:04,416 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-22 17:56:04,417 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:04,417 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:04,417 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:04,417 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:04,417 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:04,417 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:04,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:04,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:04,485 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/c19e440daae64832989d0ce83d926264 2023-03-22 17:56:04,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507764481, max timestamp=1679507764494Num uniques 859; 2023-03-22 17:56:04,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507764481, max timestamp=1679507764494Num uniques 859; 2023-03-22 17:56:04,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507764481, max timestamp=1679507764494Num uniques 859; 2023-03-22 17:56:04,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:04,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:04,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:04,510 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/fc6fa3aee12443e080b816cdcf724701 2023-03-22 17:56:04,525 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/e695c3c144fa4c50b03a9c24bb0fb637 2023-03-22 17:56:04,529 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/c19e440daae64832989d0ce83d926264 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c19e440daae64832989d0ce83d926264 2023-03-22 17:56:04,532 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c19e440daae64832989d0ce83d926264, entries=200, sequenceid=431, filesize=14.3 K 2023-03-22 17:56:04,533 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/fc6fa3aee12443e080b816cdcf724701 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fc6fa3aee12443e080b816cdcf724701 2023-03-22 17:56:04,537 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fc6fa3aee12443e080b816cdcf724701, entries=150, sequenceid=431, filesize=11.9 K 2023-03-22 17:56:04,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/e695c3c144fa4c50b03a9c24bb0fb637 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e695c3c144fa4c50b03a9c24bb0fb637 2023-03-22 17:56:04,540 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e695c3c144fa4c50b03a9c24bb0fb637, entries=150, sequenceid=431, filesize=11.9 K 2023-03-22 17:56:04,540 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 124ms, sequenceid=431, compaction requested=true 2023-03-22 17:56:04,541 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:04,541 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,541 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:04,541 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:04,541 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:04,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:04,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:04,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-22 17:56:04,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:04,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:56:04,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:04,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:56:04,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:04,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-22 17:56:04,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,549 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,549 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66e8b044[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:04,551 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:04,551 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,552 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:04,553 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,553 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:04,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,555 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:04,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:04,558 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:04,558 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:04,558 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:04,558 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:04,558 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:04,559 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,559 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,560 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,560 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,563 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,566 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:04,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:04,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,567 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:04,567 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:04,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/6be6749fe840417285fc9ee33c2687a6 2023-03-22 17:56:04,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54954 deadline: 1679507824558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507824561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507824562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54976 deadline: 1679507824564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507824564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/878948773ad8418fa1e4f86f5ba0a666 2023-03-22 17:56:04,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/485ae869a6a94be7ad98e33862f0ceff 2023-03-22 17:56:04,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/6be6749fe840417285fc9ee33c2687a6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/6be6749fe840417285fc9ee33c2687a6 2023-03-22 17:56:04,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/6be6749fe840417285fc9ee33c2687a6, entries=350, sequenceid=469, filesize=21.4 K 2023-03-22 17:56:04,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/878948773ad8418fa1e4f86f5ba0a666 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/878948773ad8418fa1e4f86f5ba0a666 2023-03-22 17:56:04,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/878948773ad8418fa1e4f86f5ba0a666, entries=150, sequenceid=469, filesize=11.9 K 2023-03-22 17:56:04,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/485ae869a6a94be7ad98e33862f0ceff as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/485ae869a6a94be7ad98e33862f0ceff 2023-03-22 17:56:04,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/485ae869a6a94be7ad98e33862f0ceff, entries=150, sequenceid=469, filesize=11.9 K 2023-03-22 17:56:04,635 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 523fd4cd7256d4e27c9367c2949e4282 in 89ms, sequenceid=469, compaction requested=true 2023-03-22 17:56:04,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:04,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:04,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:04,635 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:04,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:04,635 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:04,637 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:04,637 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:56:04,637 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,637 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 73884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:04,637 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5e8c79d4924d4dbcb08474ac3c53a070, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/4f83eb768c8d45b99728534dc94cdbd0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0f22b662b6974d5c98df9aa7feff7f3a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fc6fa3aee12443e080b816cdcf724701, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/878948773ad8418fa1e4f86f5ba0a666] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=60.2 K 2023-03-22 17:56:04,637 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:56:04,637 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,637 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/e9bd3a9074dc482d986b19a70fd5d758, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/7f56a5ded9394cac965dc49fb9c0076c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/85c94bff4e8b489482f394da09b6493b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c19e440daae64832989d0ce83d926264, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/6be6749fe840417285fc9ee33c2687a6] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=72.2 K 2023-03-22 17:56:04,638 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5e8c79d4924d4dbcb08474ac3c53a070, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679507762111 2023-03-22 17:56:04,638 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e9bd3a9074dc482d986b19a70fd5d758, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679507762111 2023-03-22 17:56:04,638 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 4f83eb768c8d45b99728534dc94cdbd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679507762440 2023-03-22 17:56:04,638 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7f56a5ded9394cac965dc49fb9c0076c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679507762440 2023-03-22 17:56:04,638 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0f22b662b6974d5c98df9aa7feff7f3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1679507763309 2023-03-22 17:56:04,638 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 85c94bff4e8b489482f394da09b6493b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1679507763309 2023-03-22 17:56:04,638 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting fc6fa3aee12443e080b816cdcf724701, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1679507764404 2023-03-22 17:56:04,640 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting c19e440daae64832989d0ce83d926264, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1679507764404 2023-03-22 17:56:04,640 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 878948773ad8418fa1e4f86f5ba0a666, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679507764533 2023-03-22 17:56:04,641 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6be6749fe840417285fc9ee33c2687a6, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679507764518 2023-03-22 17:56:04,665 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#455 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:04,667 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:04,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:04,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:04,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:04,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:04,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:04,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:04,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:04,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:04,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:04,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:04,682 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#456 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:04,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:04,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:04,688 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:04,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:04,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:04,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,689 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,690 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,690 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,691 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,691 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:04,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:04,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:04,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:04,709 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:04,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:04,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:04,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:04,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:04,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:04,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,726 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,727 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,727 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/ab3c4c9763c24ba484ce52ba97e19ace 2023-03-22 17:56:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:04,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116b8e27[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:04,731 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:04,732 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,735 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/27472a70c6ed491b943bd5795c5e28a7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/27472a70c6ed491b943bd5795c5e28a7 2023-03-22 17:56:04,736 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:04,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:04,736 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:04,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:04,738 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:04,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:04,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:04,743 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:04,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:04,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:04,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:04,743 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:04,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,746 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,747 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,748 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into 27472a70c6ed491b943bd5795c5e28a7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:04,748 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:04,748 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=11, startTime=1679507764635; duration=0sec 2023-03-22 17:56:04,748 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:04,748 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:04,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507824731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507824730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507764724, max timestamp=1679507764739Num uniques 859; 2023-03-22 17:56:04,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507824733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:04,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507764724, max timestamp=1679507764739Num uniques 859; 2023-03-22 17:56:04,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,751 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:04,751 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:56:04,751 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,751 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/9a5615ea28834320a63b10b1d27ca4a3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c66468b224c84e6185599bcb57fcc0b1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a6ab4778fca3462eae88193b8ac79083, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e695c3c144fa4c50b03a9c24bb0fb637, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/485ae869a6a94be7ad98e33862f0ceff] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=60.2 K 2023-03-22 17:56:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,751 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 9a5615ea28834320a63b10b1d27ca4a3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679507762111 2023-03-22 17:56:04,752 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c66468b224c84e6185599bcb57fcc0b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679507762440 2023-03-22 17:56:04,752 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a6ab4778fca3462eae88193b8ac79083, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1679507763309 2023-03-22 17:56:04,752 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e695c3c144fa4c50b03a9c24bb0fb637, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1679507764404 2023-03-22 17:56:04,753 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 485ae869a6a94be7ad98e33862f0ceff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679507764533 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,753 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:04,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:04,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:04,754 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:04,754 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:04,773 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#459 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:04,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507824805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507824811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,854 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:04,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:04,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:04,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507824850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:04,859 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:04,859 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,859 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507824850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,859 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:04,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507824851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,862 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:04,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:04,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:04,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:04,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:04,864 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:04,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:04,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:04,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,866 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:04,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@306910da[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:04,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:04,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,869 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:04,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:04,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:04,870 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,870 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:04,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:04,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:04,872 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:04,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:04,875 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:04,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:04,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:04,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:04,875 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:04,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,880 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:04,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:04,883 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:04,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:04,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:04,883 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:04,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:04,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:04,883 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:04,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507824917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:04,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507824923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,984 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:04,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:04,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:04,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:04,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:04,987 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:04,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:04,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:04,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,988 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,989 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:04,989 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:04,989 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,989 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:04,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:04,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:04,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:04,990 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:04,990 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:04,991 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:04,992 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:04,992 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:04,992 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,992 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:04,992 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:04,992 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:04,992 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:04,992 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:04,993 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,993 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:04,993 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:04,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,998 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:04,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:04,998 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:04,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:04,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd2e271[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:04,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:04,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,001 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,002 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:05,002 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:05,002 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:05,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:05,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:05,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,004 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:05,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:05,006 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:05,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:05,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:05,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:05,006 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:05,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,009 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,011 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:05,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:05,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,015 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,016 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:05,016 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:05,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:05,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507825059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507825060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507825061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,116 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:05,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:05,119 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/d8d4ed351b834320b5b8784dcceae0a3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d8d4ed351b834320b5b8784dcceae0a3 2023-03-22 17:56:05,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:05,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:05,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:05,128 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:05,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:05,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:05,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,130 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,130 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507825122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:05,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:05,134 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into d8d4ed351b834320b5b8784dcceae0a3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:05,134 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:05,135 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=11, startTime=1679507764635; duration=0sec 2023-03-22 17:56:05,135 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:05,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:05,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:05,139 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:05,139 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,139 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:05,139 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:05,140 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:05,140 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,140 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:05,148 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:05,148 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507825133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,149 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:05,149 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:05,149 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,149 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:05,149 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:05,152 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,152 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,153 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:05,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f1b093f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:05,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:05,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,155 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:05,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,156 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:05,162 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:05,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:05,165 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:05,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:05,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:05,165 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:05,165 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:05,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,167 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,169 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/cda2c8d84c3742dcb2848439bebd2772 2023-03-22 17:56:05,171 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:05,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,174 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:05,174 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:05,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,196 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/19ef7ad9f3d64b47aee1fe3aa6ea1aae as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/19ef7ad9f3d64b47aee1fe3aa6ea1aae 2023-03-22 17:56:05,202 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into 19ef7ad9f3d64b47aee1fe3aa6ea1aae(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:05,202 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:05,202 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=11, startTime=1679507764635; duration=0sec 2023-03-22 17:56:05,202 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:05,275 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:05,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:05,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:05,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:05,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:05,278 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:05,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:05,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:05,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,281 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:05,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:05,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:05,282 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,282 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:05,282 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:05,283 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:05,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:05,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:05,283 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:05,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,285 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62cca04d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:05,287 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:05,287 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,288 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:05,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:05,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:05,289 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:05,291 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:05,291 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,291 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,291 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:05,294 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:05,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:05,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:05,294 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:05,294 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:05,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,298 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,301 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:05,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:05,301 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,302 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:05,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:05,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,302 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:05,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,303 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,303 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,303 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,303 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,303 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,303 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,303 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507825367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507825368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507825370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,402 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:05,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:05,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:05,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:05,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:05,405 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:05,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:05,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:05,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,407 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,407 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,408 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,408 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:05,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:05,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:05,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:05,409 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,409 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:05,410 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,410 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:05,410 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:05,410 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:05,410 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:05,412 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:05,414 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:05,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1adb930e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:05,414 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,417 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:05,418 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,418 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:05,419 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:05,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:05,423 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:05,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:05,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:05,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:05,423 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:05,424 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,424 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,424 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,426 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,428 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:05,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:05,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:05,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,428 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:05,428 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:05,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507825434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507825452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,529 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:05,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:05,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:05,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:05,532 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:05,532 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,532 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:05,532 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:05,532 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,533 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,535 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,535 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,535 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,535 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:05,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:05,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:05,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:05,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:05,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:05,538 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:05,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1013b4e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:05,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:05,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:05,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:05,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:05,543 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:05,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:05,545 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:05,545 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:05,545 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:05,545 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:05,546 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:05,547 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,547 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,547 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:05,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:05,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:05,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:05,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:05,556 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:05,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:05,557 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,557 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:05,557 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:05,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:05,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/bb96a8d3ab1247b1855a1244da47456e 2023-03-22 17:56:05,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/ab3c4c9763c24ba484ce52ba97e19ace as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/ab3c4c9763c24ba484ce52ba97e19ace 2023-03-22 17:56:05,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/ab3c4c9763c24ba484ce52ba97e19ace, entries=200, sequenceid=479, filesize=14.3 K 2023-03-22 17:56:05,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/cda2c8d84c3742dcb2848439bebd2772 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/cda2c8d84c3742dcb2848439bebd2772 2023-03-22 17:56:05,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/cda2c8d84c3742dcb2848439bebd2772, entries=150, sequenceid=479, filesize=11.9 K 2023-03-22 17:56:05,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/bb96a8d3ab1247b1855a1244da47456e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb96a8d3ab1247b1855a1244da47456e 2023-03-22 17:56:05,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb96a8d3ab1247b1855a1244da47456e, entries=150, sequenceid=479, filesize=11.9 K 2023-03-22 17:56:05,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for 523fd4cd7256d4e27c9367c2949e4282 in 946ms, sequenceid=479, compaction requested=false 2023-03-22 17:56:05,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:05,657 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:05,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:05,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:05,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:05,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:05,660 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:05,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:05,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:05,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,662 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:05,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:05,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:05,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:05,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:05,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:05,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:05,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:05,670 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:05,670 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:05,670 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:05,670 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:05,670 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=131.94 KB heapSize=383.31 KB 2023-03-22 17:56:05,671 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:05,671 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:05,671 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:05,671 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:05,671 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:05,671 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:05,679 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/d71f170926d441e7b219e7bd62dc1cf9 2023-03-22 17:56:05,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:05,885 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:05,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507825894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507825895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507825896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507825946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:05,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:05,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507825963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507825997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507826001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507826001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,094 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/0ed553bebe7647b9b1c40b7c8ffb5b13 2023-03-22 17:56:06,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507826204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507826207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507826208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,504 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/f90110bc25a24239bcc8edbf1f784e90 2023-03-22 17:56:06,512 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/d71f170926d441e7b219e7bd62dc1cf9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d71f170926d441e7b219e7bd62dc1cf9 2023-03-22 17:56:06,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507826512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,516 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d71f170926d441e7b219e7bd62dc1cf9, entries=150, sequenceid=514, filesize=11.9 K 2023-03-22 17:56:06,517 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/0ed553bebe7647b9b1c40b7c8ffb5b13 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0ed553bebe7647b9b1c40b7c8ffb5b13 2023-03-22 17:56:06,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507826514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:06,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507826516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,521 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0ed553bebe7647b9b1c40b7c8ffb5b13, entries=150, sequenceid=514, filesize=11.9 K 2023-03-22 17:56:06,522 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/f90110bc25a24239bcc8edbf1f784e90 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f90110bc25a24239bcc8edbf1f784e90 2023-03-22 17:56:06,535 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f90110bc25a24239bcc8edbf1f784e90, entries=150, sequenceid=514, filesize=11.9 K 2023-03-22 17:56:06,535 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~383.27 KB/392464, currentSize=46.96 KB/48090 for 523fd4cd7256d4e27c9367c2949e4282 in 865ms, sequenceid=514, compaction requested=true 2023-03-22 17:56:06,535 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:06,535 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:06,535 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:06,535 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:06,536 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,536 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:06,536 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:06,537 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:06,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:06,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:06,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:06,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a498a0[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:06,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:06,541 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:06,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:06,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:06,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:06,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:06,542 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:06,543 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,543 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:06,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:06,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:06,544 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:06,545 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:06,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:06,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:06,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:06,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:06,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:06,547 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:06,547 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:06,547 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:06,547 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:06,547 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:06,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:06,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:06,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:06,550 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:06,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:06,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:06,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:06,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:06,555 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:06,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:06,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:06,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:06,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:06,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:06,556 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:06,556 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:06,656 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:06,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:06,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:06,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:06,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:06,660 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:06,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:06,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:06,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:06,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,660 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:06,662 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:06,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:06,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:06,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:06,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:06,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:06,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:06,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:06,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:06,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:06,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:06,664 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:06,664 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:06,664 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:06,664 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:06,664 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:06,665 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:06,665 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:06,665 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:06,665 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:06,665 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:06,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:06,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:07,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507766982, max timestamp=1679507767015Num uniques 859; 2023-03-22 17:56:07,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:07,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507766982, max timestamp=1679507767015Num uniques 859; 2023-03-22 17:56:07,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507766982, max timestamp=1679507767015Num uniques 859; 2023-03-22 17:56:07,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:07,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:07,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507827045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507827054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507827054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507827055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507827056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,073 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/fe951b11ed2f4f9d9743fd7dc614529a 2023-03-22 17:56:07,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507827167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507827167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507827167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507827168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507827168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507827375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507827375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507827375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507827375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507827375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,482 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/9b8eba02f31445239341ac39cb9b5534 2023-03-22 17:56:07,492 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/0f778d2dcef84acc8a6c9c026f0efdbe 2023-03-22 17:56:07,495 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/fe951b11ed2f4f9d9743fd7dc614529a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fe951b11ed2f4f9d9743fd7dc614529a 2023-03-22 17:56:07,499 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fe951b11ed2f4f9d9743fd7dc614529a, entries=150, sequenceid=524, filesize=11.9 K 2023-03-22 17:56:07,500 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/9b8eba02f31445239341ac39cb9b5534 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9b8eba02f31445239341ac39cb9b5534 2023-03-22 17:56:07,503 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9b8eba02f31445239341ac39cb9b5534, entries=150, sequenceid=524, filesize=11.9 K 2023-03-22 17:56:07,503 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/0f778d2dcef84acc8a6c9c026f0efdbe as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0f778d2dcef84acc8a6c9c026f0efdbe 2023-03-22 17:56:07,507 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0f778d2dcef84acc8a6c9c026f0efdbe, entries=150, sequenceid=524, filesize=11.9 K 2023-03-22 17:56:07,507 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 843ms, sequenceid=524, compaction requested=true 2023-03-22 17:56:07,507 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:07,507 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:07,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,510 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:07,510 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:07,511 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ed80f1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:07,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:07,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:07,513 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:07,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:07,513 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:07,513 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:07,513 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:07,514 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,514 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:07,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:07,515 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:07,515 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:07,517 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:07,517 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:07,517 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:07,517 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:07,517 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:07,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:07,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,519 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:07,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:07,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:07,521 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:07,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:07,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:07,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:07,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:07,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:07,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:07,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:07,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:07,524 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:07,525 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,525 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:07,525 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:07,525 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:07,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:07,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:07,525 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,625 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:07,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:07,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:07,628 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:07,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:07,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:07,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:07,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:07,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:07,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:07,631 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,631 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:07,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:07,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:07,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:07,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:07,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:07,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:07,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:07,633 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:07,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:07,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:07,633 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-22 17:56:07,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:07,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:07,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:07,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:07,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:07,633 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:07,640 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/fbffa8127bfd45f9bc6413bc0311e79c 2023-03-22 17:56:07,650 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/6a98b00d68fa459cb59938ebf1fe7cc3 2023-03-22 17:56:07,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:07,688 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:07,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507827690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507827690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507827691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507827699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507827700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507827801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507827801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507827802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507827808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:07,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:07,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507827808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507828006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507828006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507828007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507828015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507828015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,070 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/de73afa187e8474a852e88461903895d 2023-03-22 17:56:08,073 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/fbffa8127bfd45f9bc6413bc0311e79c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fbffa8127bfd45f9bc6413bc0311e79c 2023-03-22 17:56:08,078 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fbffa8127bfd45f9bc6413bc0311e79c, entries=150, sequenceid=561, filesize=11.9 K 2023-03-22 17:56:08,079 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/6a98b00d68fa459cb59938ebf1fe7cc3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6a98b00d68fa459cb59938ebf1fe7cc3 2023-03-22 17:56:08,082 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6a98b00d68fa459cb59938ebf1fe7cc3, entries=150, sequenceid=561, filesize=11.9 K 2023-03-22 17:56:08,083 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/de73afa187e8474a852e88461903895d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de73afa187e8474a852e88461903895d 2023-03-22 17:56:08,086 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de73afa187e8474a852e88461903895d, entries=150, sequenceid=561, filesize=11.9 K 2023-03-22 17:56:08,087 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 523fd4cd7256d4e27c9367c2949e4282 in 454ms, sequenceid=561, compaction requested=true 2023-03-22 17:56:08,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:08,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:08,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:08,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:08,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:08,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:08,088 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:08,089 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:08,089 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:08,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:08,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37682e21[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:08,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:08,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:08,093 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:08,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:08,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:08,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:08,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:08,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,093 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:08,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:08,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:08,098 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,098 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:08,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:08,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:08,101 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:08,101 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:08,101 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:08,101 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:08,101 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:08,102 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,103 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:08,103 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:08,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:08,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:08,109 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:08,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:08,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:08,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:08,110 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:08,110 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:08,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:08,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:08,110 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:08,110 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:08,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:08,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:08,210 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:08,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:08,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:08,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:08,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:08,213 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:08,213 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:08,213 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:08,213 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:08,214 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,214 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:08,216 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:08,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:08,217 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,217 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:08,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:08,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:08,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:08,218 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:08,218 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:08,218 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:08,218 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:08,218 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:08,218 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:08,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:08,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:08,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:08,229 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/a22ba1a151304b2a8b25118637bfc571 2023-03-22 17:56:08,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:08,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:08,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507768340, max timestamp=1679507768355Num uniques 859; 2023-03-22 17:56:08,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:08,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507768340, max timestamp=1679507768355Num uniques 859; 2023-03-22 17:56:08,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507768340, max timestamp=1679507768355Num uniques 859; 2023-03-22 17:56:08,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:08,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:08,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507828386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507828387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507828387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507828395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507828398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507828499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507828499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507828499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507828505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507828508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,651 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/62fd9ed54dd0443fab32087f343e41b4 2023-03-22 17:56:08,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507828708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507828709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507828710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507828713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:08,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:08,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507828718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507829012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507829014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507829019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507829023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507829027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,060 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/8277233faf3b4fb7988e543a74005ae1 2023-03-22 17:56:09,063 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/a22ba1a151304b2a8b25118637bfc571 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a22ba1a151304b2a8b25118637bfc571 2023-03-22 17:56:09,067 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a22ba1a151304b2a8b25118637bfc571, entries=100, sequenceid=571, filesize=9.5 K 2023-03-22 17:56:09,067 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/62fd9ed54dd0443fab32087f343e41b4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/62fd9ed54dd0443fab32087f343e41b4 2023-03-22 17:56:09,071 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/62fd9ed54dd0443fab32087f343e41b4, entries=100, sequenceid=571, filesize=9.5 K 2023-03-22 17:56:09,071 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/8277233faf3b4fb7988e543a74005ae1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/8277233faf3b4fb7988e543a74005ae1 2023-03-22 17:56:09,074 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/8277233faf3b4fb7988e543a74005ae1, entries=100, sequenceid=571, filesize=9.5 K 2023-03-22 17:56:09,075 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 857ms, sequenceid=571, compaction requested=true 2023-03-22 17:56:09,075 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:09,075 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:09,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:09,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:09,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:09,075 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:09,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,077 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,077 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,077 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fbba8c9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:09,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:09,079 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,080 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:09,080 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,080 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:09,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,081 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:09,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,081 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:09,081 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:09,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:09,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:09,083 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:09,083 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:09,083 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:09,083 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:09,083 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:09,085 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,085 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,085 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:09,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,086 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:09,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:09,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,088 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,090 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:09,090 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:09,090 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,090 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,090 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:09,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:09,090 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:09,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:09,090 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,091 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,090 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:09,091 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:09,091 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,091 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:09,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:09,191 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:09,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:09,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:09,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:09,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:09,194 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:09,194 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:09,194 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:09,194 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:09,195 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,195 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:09,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:09,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:09,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:09,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:09,197 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:09,197 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:09,198 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-22 17:56:09,198 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:09,198 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:09,198 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:09,198 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:09,198 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:09,198 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:09,205 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/9c823d4e5b3a4be2ade6497b9e210696 2023-03-22 17:56:09,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:09,531 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:09,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507829537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507829542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507829542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507829544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507829544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,618 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/e2fe5cd8b3534691b7a48e1021b9d52b 2023-03-22 17:56:09,628 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/69f89b453eb84540a722d3f7842be09d 2023-03-22 17:56:09,631 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/9c823d4e5b3a4be2ade6497b9e210696 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9c823d4e5b3a4be2ade6497b9e210696 2023-03-22 17:56:09,635 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9c823d4e5b3a4be2ade6497b9e210696, entries=150, sequenceid=607, filesize=11.9 K 2023-03-22 17:56:09,636 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/e2fe5cd8b3534691b7a48e1021b9d52b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/e2fe5cd8b3534691b7a48e1021b9d52b 2023-03-22 17:56:09,639 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/e2fe5cd8b3534691b7a48e1021b9d52b, entries=150, sequenceid=607, filesize=11.9 K 2023-03-22 17:56:09,640 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/69f89b453eb84540a722d3f7842be09d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/69f89b453eb84540a722d3f7842be09d 2023-03-22 17:56:09,643 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/69f89b453eb84540a722d3f7842be09d, entries=150, sequenceid=607, filesize=11.9 K 2023-03-22 17:56:09,644 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 523fd4cd7256d4e27c9367c2949e4282 in 446ms, sequenceid=607, compaction requested=true 2023-03-22 17:56:09,644 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:09,644 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:09,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:09,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:09,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:09,644 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:09,646 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,646 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,647 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,647 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d084606[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:09,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:09,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,650 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:09,650 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,651 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:09,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:09,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:09,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:09,654 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:09,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:09,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:09,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:09,654 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:09,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:09,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-22 17:56:09,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:09,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:09,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:09,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:09,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:09,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:09,657 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,657 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,657 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:09,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:09,658 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,659 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:09,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:09,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:09,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:09,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:09,661 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,662 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:09,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:09,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:09,662 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:09,662 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:09,662 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,662 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507829679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507829680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507829684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507829685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507829693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,762 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:09,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:09,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:09,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:09,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:09,766 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:09,766 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:09,767 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:09,767 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:09,767 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,767 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,770 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:09,770 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:09,770 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,770 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:09,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:09,771 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:09,771 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:09,771 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:09,772 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:09,772 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:09,772 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:09,772 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:09,772 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:09,773 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,773 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,774 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,774 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ffa95e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:09,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:09,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:09,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:09,778 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,778 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:09,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:09,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,779 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:09,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:09,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:09,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:09,782 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:09,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:09,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:09,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:09,782 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:09,784 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,785 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,785 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:09,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,786 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,787 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:09,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:09,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:09,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:09,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:09,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:09,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:09,790 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:09,790 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:09,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:09,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507829794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507829794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507829795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507829797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:09,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507829797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,890 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:09,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:09,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:09,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:09,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:09,894 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:09,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:09,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:09,894 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:09,895 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,895 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,899 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,899 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:09,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:09,899 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,899 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:09,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:09,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:09,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:09,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:09,901 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:09,901 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:09,901 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:09,901 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:09,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:09,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:09,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:09,901 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:09,903 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,903 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:09,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:09,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6109846[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:09,907 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:09,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:09,908 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:09,909 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:09,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:09,911 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:09,911 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:09,911 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:09,911 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:09,911 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:09,914 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,914 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,914 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:09,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:09,915 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,916 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:09,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:09,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:09,920 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:09,920 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:09,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:09,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:09,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:09,921 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:09,921 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:09,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:09,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:09,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:09,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507830001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507830002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507830003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507830004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507830005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,021 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:10,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:10,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:10,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:10,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:10,024 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:10,024 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,024 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:10,024 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:10,025 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,025 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,026 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,026 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,027 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,027 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:10,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:10,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:10,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:10,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:10,028 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:10,028 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,028 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:10,028 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:10,029 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:10,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:10,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:10,029 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:10,031 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,031 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,031 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,031 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:10,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648f95ad[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:10,033 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:10,034 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,035 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:10,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,035 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:10,037 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,037 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:10,038 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,038 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:10,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:10,040 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:10,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:10,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:10,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:10,040 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:10,042 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,042 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,042 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:10,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,043 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,044 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,046 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,046 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,046 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,046 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,046 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:10,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:10,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:10,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:10,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:10,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:10,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,047 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:10,047 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:10,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,047 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/19fcf2e3d3c64abe99255677bd5f32e4 2023-03-22 17:56:10,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/214e1c6374004adaae1fddf5e0d055e7 2023-03-22 17:56:10,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/cc3496a0c74d497aa1bcf80bbe0b6218 2023-03-22 17:56:10,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/19fcf2e3d3c64abe99255677bd5f32e4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/19fcf2e3d3c64abe99255677bd5f32e4 2023-03-22 17:56:10,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/19fcf2e3d3c64abe99255677bd5f32e4, entries=150, sequenceid=619, filesize=11.9 K 2023-03-22 17:56:10,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/214e1c6374004adaae1fddf5e0d055e7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/214e1c6374004adaae1fddf5e0d055e7 2023-03-22 17:56:10,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/214e1c6374004adaae1fddf5e0d055e7, entries=150, sequenceid=619, filesize=11.9 K 2023-03-22 17:56:10,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/cc3496a0c74d497aa1bcf80bbe0b6218 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/cc3496a0c74d497aa1bcf80bbe0b6218 2023-03-22 17:56:10,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/cc3496a0c74d497aa1bcf80bbe0b6218, entries=150, sequenceid=619, filesize=11.9 K 2023-03-22 17:56:10,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 523fd4cd7256d4e27c9367c2949e4282 in 444ms, sequenceid=619, compaction requested=true 2023-03-22 17:56:10,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:10,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:10,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:10,100 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-22 17:56:10,100 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-22 17:56:10,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-22 17:56:10,102 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95964 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-22 17:56:10,102 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98404 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-22 17:56:10,102 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:56:10,102 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:56:10,102 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,102 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,102 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d8d4ed351b834320b5b8784dcceae0a3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/ab3c4c9763c24ba484ce52ba97e19ace, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d71f170926d441e7b219e7bd62dc1cf9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fe951b11ed2f4f9d9743fd7dc614529a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fbffa8127bfd45f9bc6413bc0311e79c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a22ba1a151304b2a8b25118637bfc571, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9c823d4e5b3a4be2ade6497b9e210696, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/19fcf2e3d3c64abe99255677bd5f32e4] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=96.1 K 2023-03-22 17:56:10,102 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/27472a70c6ed491b943bd5795c5e28a7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/cda2c8d84c3742dcb2848439bebd2772, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0ed553bebe7647b9b1c40b7c8ffb5b13, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9b8eba02f31445239341ac39cb9b5534, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6a98b00d68fa459cb59938ebf1fe7cc3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/62fd9ed54dd0443fab32087f343e41b4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/e2fe5cd8b3534691b7a48e1021b9d52b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/214e1c6374004adaae1fddf5e0d055e7] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=93.7 K 2023-03-22 17:56:10,102 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d8d4ed351b834320b5b8784dcceae0a3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679507764533 2023-03-22 17:56:10,103 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 27472a70c6ed491b943bd5795c5e28a7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679507764533 2023-03-22 17:56:10,103 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting ab3c4c9763c24ba484ce52ba97e19ace, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1679507764552 2023-03-22 17:56:10,103 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting cda2c8d84c3742dcb2848439bebd2772, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1679507764559 2023-03-22 17:56:10,103 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d71f170926d441e7b219e7bd62dc1cf9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1679507764791 2023-03-22 17:56:10,103 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0ed553bebe7647b9b1c40b7c8ffb5b13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1679507764791 2023-03-22 17:56:10,104 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting fe951b11ed2f4f9d9743fd7dc614529a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679507765886 2023-03-22 17:56:10,104 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9b8eba02f31445239341ac39cb9b5534, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679507765886 2023-03-22 17:56:10,104 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting fbffa8127bfd45f9bc6413bc0311e79c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1679507767044 2023-03-22 17:56:10,104 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6a98b00d68fa459cb59938ebf1fe7cc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1679507767044 2023-03-22 17:56:10,104 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a22ba1a151304b2a8b25118637bfc571, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1679507767688 2023-03-22 17:56:10,104 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 62fd9ed54dd0443fab32087f343e41b4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1679507767688 2023-03-22 17:56:10,105 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 9c823d4e5b3a4be2ade6497b9e210696, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1679507768367 2023-03-22 17:56:10,105 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e2fe5cd8b3534691b7a48e1021b9d52b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1679507768367 2023-03-22 17:56:10,105 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 19fcf2e3d3c64abe99255677bd5f32e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1679507769542 2023-03-22 17:56:10,105 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 214e1c6374004adaae1fddf5e0d055e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1679507769542 2023-03-22 17:56:10,117 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:10,126 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#480 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:10,147 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:10,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:10,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:10,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:10,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:10,150 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:10,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:10,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:10,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,152 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,152 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,153 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,153 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:10,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:10,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:10,154 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:10,154 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,154 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:10,154 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:10,154 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:10,154 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,154 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:10,154 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-22 17:56:10,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:10,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:10,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:10,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:10,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:10,155 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:10,163 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/913b665a6a824e4d871b5971e46f203a 2023-03-22 17:56:10,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:10,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:10,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507830326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507830326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507830327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507830328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507830337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507830438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507830438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507830438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507830438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507830445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,533 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/64378d9d76f34b3aacf4820b32091b8a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/64378d9d76f34b3aacf4820b32091b8a 2023-03-22 17:56:10,537 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into 64378d9d76f34b3aacf4820b32091b8a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:10,538 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:10,538 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=8, startTime=1679507770100; duration=0sec 2023-03-22 17:56:10,538 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:10,538 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-22 17:56:10,541 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95930 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-22 17:56:10,541 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:56:10,542 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,542 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/19ef7ad9f3d64b47aee1fe3aa6ea1aae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb96a8d3ab1247b1855a1244da47456e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f90110bc25a24239bcc8edbf1f784e90, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0f778d2dcef84acc8a6c9c026f0efdbe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de73afa187e8474a852e88461903895d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/8277233faf3b4fb7988e543a74005ae1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/69f89b453eb84540a722d3f7842be09d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/cc3496a0c74d497aa1bcf80bbe0b6218] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=93.7 K 2023-03-22 17:56:10,542 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/6527d8b7b6fa40f0a598681d55aa249d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6527d8b7b6fa40f0a598681d55aa249d 2023-03-22 17:56:10,542 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 19ef7ad9f3d64b47aee1fe3aa6ea1aae, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679507764533 2023-03-22 17:56:10,543 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting bb96a8d3ab1247b1855a1244da47456e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1679507764559 2023-03-22 17:56:10,543 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f90110bc25a24239bcc8edbf1f784e90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1679507764791 2023-03-22 17:56:10,544 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0f778d2dcef84acc8a6c9c026f0efdbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679507765886 2023-03-22 17:56:10,544 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting de73afa187e8474a852e88461903895d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1679507767044 2023-03-22 17:56:10,545 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 8277233faf3b4fb7988e543a74005ae1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1679507767688 2023-03-22 17:56:10,545 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 69f89b453eb84540a722d3f7842be09d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1679507768367 2023-03-22 17:56:10,545 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting cc3496a0c74d497aa1bcf80bbe0b6218, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1679507769542 2023-03-22 17:56:10,546 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into 6527d8b7b6fa40f0a598681d55aa249d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:10,546 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:10,547 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=8, startTime=1679507770100; duration=0sec 2023-03-22 17:56:10,547 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:10,557 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:10,576 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/d8c897d22a8747f6bce1615ad7e9d141 2023-03-22 17:56:10,584 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/0c4dad6292d340d7ace470c24f4bb140 2023-03-22 17:56:10,588 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/913b665a6a824e4d871b5971e46f203a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/913b665a6a824e4d871b5971e46f203a 2023-03-22 17:56:10,590 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/913b665a6a824e4d871b5971e46f203a, entries=150, sequenceid=639, filesize=11.9 K 2023-03-22 17:56:10,591 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/d8c897d22a8747f6bce1615ad7e9d141 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/d8c897d22a8747f6bce1615ad7e9d141 2023-03-22 17:56:10,594 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/d8c897d22a8747f6bce1615ad7e9d141, entries=150, sequenceid=639, filesize=11.9 K 2023-03-22 17:56:10,595 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/0c4dad6292d340d7ace470c24f4bb140 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c4dad6292d340d7ace470c24f4bb140 2023-03-22 17:56:10,598 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c4dad6292d340d7ace470c24f4bb140, entries=150, sequenceid=639, filesize=11.9 K 2023-03-22 17:56:10,598 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 523fd4cd7256d4e27c9367c2949e4282 in 444ms, sequenceid=639, compaction requested=false 2023-03-22 17:56:10,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:10,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:10,603 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,603 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:10,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b98c74c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:10,605 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:10,605 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:10,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:10,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:10,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:10,608 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:10,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:10,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:10,612 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:10,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:10,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:10,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:10,613 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:10,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,625 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:10,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,627 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:10,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,630 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:10,630 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:10,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:10,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-22 17:56:10,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:10,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:10,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:10,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:10,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:10,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:10,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/3b34c7f2e4f040918f6e41514bc91818 2023-03-22 17:56:10,673 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/f06fa8d3e2b54493b351c91dbc166e99 2023-03-22 17:56:10,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507830680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507830681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507830682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507830688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507830688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,731 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:10,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:10,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:10,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:10,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:10,735 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:10,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:10,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:10,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,736 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,737 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:10,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:10,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:10,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:10,738 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:10,738 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:10,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:10,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:10,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:10,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a03238[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:10,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:10,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,747 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:10,747 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:10,749 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:10,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:10,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:10,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:10,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:10,751 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:10,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:10,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:10,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:10,751 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:10,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:10,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:10,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,754 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,755 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:10,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,757 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:10,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:10,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:10,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,758 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,758 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,758 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,758 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,758 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:10,758 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:10,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:10,758 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:10,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507830789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507830796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507830796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507830802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:10,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507830802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,858 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:10,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:10,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:10,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:10,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:10,861 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:10,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:10,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:10,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,864 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:10,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:10,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:10,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:10,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:10,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:10,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:10,866 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:10,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:10,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,868 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:10,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:10,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44704002[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:10,870 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:10,870 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,871 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:10,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:10,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:10,873 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:10,873 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:10,873 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:10,878 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:10,881 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:10,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:10,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:10,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:10,881 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:10,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:10,888 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,889 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,892 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:10,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,892 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:10,892 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:10,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:10,892 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:10,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:10,966 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/36efba515a694338a75a790b0ca33e14 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/36efba515a694338a75a790b0ca33e14 2023-03-22 17:56:10,970 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into 36efba515a694338a75a790b0ca33e14(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:10,970 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:10,970 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=8, startTime=1679507770100; duration=0sec 2023-03-22 17:56:10,971 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:10,993 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:10,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:10,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:10,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:10,996 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:10,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:10,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:10,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,996 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,997 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:10,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:10,998 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,998 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:10,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:10,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:10,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:10,999 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:11,001 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:11,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cca97dc[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:11,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507830997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:11,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,006 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:11,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:11,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:11,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,006 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:11,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:11,010 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:11,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:11,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:11,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:11,010 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:11,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507831004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507831005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507831008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507831008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:11,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:11,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,016 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:11,016 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:11,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:11,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ea6272bcda6b4d8f9db9b181b2eac307 2023-03-22 17:56:11,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/3b34c7f2e4f040918f6e41514bc91818 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/3b34c7f2e4f040918f6e41514bc91818 2023-03-22 17:56:11,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/3b34c7f2e4f040918f6e41514bc91818, entries=200, sequenceid=654, filesize=14.3 K 2023-03-22 17:56:11,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/f06fa8d3e2b54493b351c91dbc166e99 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f06fa8d3e2b54493b351c91dbc166e99 2023-03-22 17:56:11,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f06fa8d3e2b54493b351c91dbc166e99, entries=150, sequenceid=654, filesize=11.9 K 2023-03-22 17:56:11,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ea6272bcda6b4d8f9db9b181b2eac307 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ea6272bcda6b4d8f9db9b181b2eac307 2023-03-22 17:56:11,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ea6272bcda6b4d8f9db9b181b2eac307, entries=150, sequenceid=654, filesize=11.9 K 2023-03-22 17:56:11,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 523fd4cd7256d4e27c9367c2949e4282 in 462ms, sequenceid=654, compaction requested=true 2023-03-22 17:56:11,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,114 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:11,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:11,115 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:11,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:11,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:11,116 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:11,116 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:56:11,116 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:11,116 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:11,116 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:56:11,116 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,116 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,116 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6527d8b7b6fa40f0a598681d55aa249d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/d8c897d22a8747f6bce1615ad7e9d141, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f06fa8d3e2b54493b351c91dbc166e99] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=36.9 K 2023-03-22 17:56:11,116 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/64378d9d76f34b3aacf4820b32091b8a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/913b665a6a824e4d871b5971e46f203a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/3b34c7f2e4f040918f6e41514bc91818] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=39.2 K 2023-03-22 17:56:11,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:11,117 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6527d8b7b6fa40f0a598681d55aa249d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1679507769542 2023-03-22 17:56:11,117 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 64378d9d76f34b3aacf4820b32091b8a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1679507769542 2023-03-22 17:56:11,118 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d8c897d22a8747f6bce1615ad7e9d141, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1679507769679 2023-03-22 17:56:11,118 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 913b665a6a824e4d871b5971e46f203a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1679507769679 2023-03-22 17:56:11,118 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 3b34c7f2e4f040918f6e41514bc91818, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679507770325 2023-03-22 17:56:11,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:11,118 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f06fa8d3e2b54493b351c91dbc166e99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679507770325 2023-03-22 17:56:11,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:11,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:11,120 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:11,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:11,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:11,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,124 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,124 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,124 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,124 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:11,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:11,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:11,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:11,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:11,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,125 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:11,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:11,126 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:11,126 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,126 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:11,126 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-22 17:56:11,126 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:11,126 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:11,126 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:11,127 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:11,127 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:11,127 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:11,133 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:11,137 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:11,157 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/d851a24d43ec408e9d75af72f47a5ced 2023-03-22 17:56:11,171 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/ce0575d00a4e49f2a8ad8e6ca9be3571 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/ce0575d00a4e49f2a8ad8e6ca9be3571 2023-03-22 17:56:11,177 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into ce0575d00a4e49f2a8ad8e6ca9be3571(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:11,177 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,177 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=13, startTime=1679507771115; duration=0sec 2023-03-22 17:56:11,177 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:11,177 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:11,178 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/19252d7d8d084a6da728d26f7c107483 2023-03-22 17:56:11,179 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:11,179 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:56:11,179 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,179 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/36efba515a694338a75a790b0ca33e14, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c4dad6292d340d7ace470c24f4bb140, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ea6272bcda6b4d8f9db9b181b2eac307] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=36.8 K 2023-03-22 17:56:11,180 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 36efba515a694338a75a790b0ca33e14, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1679507769542 2023-03-22 17:56:11,180 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0c4dad6292d340d7ace470c24f4bb140, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1679507769679 2023-03-22 17:56:11,180 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ea6272bcda6b4d8f9db9b181b2eac307, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679507770325 2023-03-22 17:56:11,189 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#492 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:11,215 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ca51fe62c52b4bd19313415f7e411482 2023-03-22 17:56:11,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/d851a24d43ec408e9d75af72f47a5ced as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d851a24d43ec408e9d75af72f47a5ced 2023-03-22 17:56:11,223 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d851a24d43ec408e9d75af72f47a5ced, entries=150, sequenceid=674, filesize=11.9 K 2023-03-22 17:56:11,223 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/19252d7d8d084a6da728d26f7c107483 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/19252d7d8d084a6da728d26f7c107483 2023-03-22 17:56:11,226 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/19252d7d8d084a6da728d26f7c107483, entries=150, sequenceid=674, filesize=11.9 K 2023-03-22 17:56:11,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ca51fe62c52b4bd19313415f7e411482 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca51fe62c52b4bd19313415f7e411482 2023-03-22 17:56:11,231 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca51fe62c52b4bd19313415f7e411482, entries=150, sequenceid=674, filesize=11.9 K 2023-03-22 17:56:11,232 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 523fd4cd7256d4e27c9367c2949e4282 in 106ms, sequenceid=674, compaction requested=false 2023-03-22 17:56:11,232 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,232 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:11,234 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,234 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,234 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,234 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:11,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@516589ae[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:11,236 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:11,236 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:11,237 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,238 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:11,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,243 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,243 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,243 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:11,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:11,245 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:11,245 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:11,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:11,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:11,246 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:11,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,248 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,252 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:11,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:11,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,252 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:11,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,252 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:11,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:11,344 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:11,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:11,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:11,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:11,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:11,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:11,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:11,353 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:11,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:11,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:11,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:11,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:11,357 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:11,357 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,357 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:11,357 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:11,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,360 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,360 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:11,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:11,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:11,361 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,361 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:11,361 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:11,361 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,362 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:11,362 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:11,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16135048[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:11,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:11,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,367 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,367 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:11,367 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:11,367 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:11,367 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,367 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:11,368 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,368 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,368 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:11,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:11,371 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:11,371 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:11,371 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:11,371 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:11,371 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:11,373 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,373 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,373 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,373 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:11,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:11,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:11,378 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,378 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,378 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:11,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507771363, max timestamp=1679507771381Num uniques 859; 2023-03-22 17:56:11,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:56:11,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507771363, max timestamp=1679507771381Num uniques 859; 2023-03-22 17:56:11,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:56:11,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507771363, max timestamp=1679507771381Num uniques 859; 2023-03-22 17:56:11,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:56:11,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507831405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507831406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507831407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507831407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507831413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,479 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:11,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:11,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:11,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:11,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:11,482 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:11,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:11,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:11,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,484 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:11,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:11,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:11,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:11,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:11,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,485 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:11,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:11,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:11,486 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,486 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:11,486 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:11,486 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:11,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:11,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:11,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:11,487 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,487 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:11,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a562d8[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:11,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:11,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,491 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:11,491 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:11,491 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:11,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:11,498 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:11,501 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:11,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:11,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:11,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:11,501 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:11,503 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,503 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,503 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,503 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,505 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:11,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:11,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:11,507 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,507 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,507 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:11,507 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,507 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:11,507 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507831514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507831515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507831515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507831516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507831517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,563 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/d22e5d29b3c74c42b28da0a5b0d62070 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d22e5d29b3c74c42b28da0a5b0d62070 2023-03-22 17:56:11,566 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into d22e5d29b3c74c42b28da0a5b0d62070(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:11,567 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,567 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=13, startTime=1679507771114; duration=0sec 2023-03-22 17:56:11,567 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:11,602 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/de7176b532804a67b4aa2decef78281c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de7176b532804a67b4aa2decef78281c 2023-03-22 17:56:11,606 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into de7176b532804a67b4aa2decef78281c(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:11,606 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,606 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=13, startTime=1679507771115; duration=0sec 2023-03-22 17:56:11,606 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:11,607 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:11,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:11,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:11,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:11,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:11,610 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:11,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:11,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:11,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,612 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:11,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:11,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:11,613 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:11,613 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,613 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:11,613 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:11,613 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:11,614 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,614 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:11,614 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:11,614 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:11,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:11,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:11,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:11,617 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,618 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:11,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58596b67[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:11,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:11,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,620 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:11,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:11,622 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:11,622 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,622 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:11,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:11,624 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:11,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:11,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:11,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:11,624 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:11,627 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,627 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,628 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,629 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,635 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:11,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:11,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,636 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,636 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:11,636 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:11,636 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,637 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507831724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507831724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507831724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507831725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:11,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507831725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,736 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:11,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:11,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:11,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:11,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:11,739 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:11,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:11,739 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:11,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:11,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:11,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:11,743 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,743 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:11,743 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:11,743 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:11,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:11,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:11,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:11,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@258ca13b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:11,749 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:11,749 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:11,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:11,750 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:11,750 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,750 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:11,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:11,755 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:11,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/a9df7b58eebf41439d992a089eb568c3 2023-03-22 17:56:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:11,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:11,759 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:11,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:11,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:11,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:11,759 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:11,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/1bed71c9261f4021aaef7b6d4b732029 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:11,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:11,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,772 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,772 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,773 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:11,773 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:11,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/a01c5b4a1d294d04a9b5073b9fcb53d8 2023-03-22 17:56:11,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/a9df7b58eebf41439d992a089eb568c3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a9df7b58eebf41439d992a089eb568c3 2023-03-22 17:56:11,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a9df7b58eebf41439d992a089eb568c3, entries=150, sequenceid=686, filesize=11.9 K 2023-03-22 17:56:11,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/1bed71c9261f4021aaef7b6d4b732029 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1bed71c9261f4021aaef7b6d4b732029 2023-03-22 17:56:11,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1bed71c9261f4021aaef7b6d4b732029, entries=150, sequenceid=686, filesize=11.9 K 2023-03-22 17:56:11,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/a01c5b4a1d294d04a9b5073b9fcb53d8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a01c5b4a1d294d04a9b5073b9fcb53d8 2023-03-22 17:56:11,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a01c5b4a1d294d04a9b5073b9fcb53d8, entries=150, sequenceid=686, filesize=11.9 K 2023-03-22 17:56:11,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 451ms, sequenceid=686, compaction requested=true 2023-03-22 17:56:11,795 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:11,796 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:11,796 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:11,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:11,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:11,797 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:11,797 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:11,797 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:56:11,797 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:56:11,797 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,797 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,797 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d22e5d29b3c74c42b28da0a5b0d62070, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d851a24d43ec408e9d75af72f47a5ced, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a9df7b58eebf41439d992a089eb568c3] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=37.0 K 2023-03-22 17:56:11,797 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/ce0575d00a4e49f2a8ad8e6ca9be3571, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/19252d7d8d084a6da728d26f7c107483, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1bed71c9261f4021aaef7b6d4b732029] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=37.0 K 2023-03-22 17:56:11,797 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d22e5d29b3c74c42b28da0a5b0d62070, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679507770325 2023-03-22 17:56:11,797 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ce0575d00a4e49f2a8ad8e6ca9be3571, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679507770325 2023-03-22 17:56:11,798 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d851a24d43ec408e9d75af72f47a5ced, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1679507770679 2023-03-22 17:56:11,798 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 19252d7d8d084a6da728d26f7c107483, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1679507770679 2023-03-22 17:56:11,798 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a9df7b58eebf41439d992a089eb568c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1679507771317 2023-03-22 17:56:11,798 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1bed71c9261f4021aaef7b6d4b732029, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1679507771317 2023-03-22 17:56:11,804 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:11,805 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#498 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:11,821 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/9dd972dea5e64c9195abb6513cecbeb8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9dd972dea5e64c9195abb6513cecbeb8 2023-03-22 17:56:11,824 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/f781447dc0e34fb5b9921123c593058d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f781447dc0e34fb5b9921123c593058d 2023-03-22 17:56:11,827 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into 9dd972dea5e64c9195abb6513cecbeb8(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:11,827 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,827 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=13, startTime=1679507771796; duration=0sec 2023-03-22 17:56:11,827 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:11,827 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:11,827 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into f781447dc0e34fb5b9921123c593058d(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:11,827 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,828 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=13, startTime=1679507771796; duration=0sec 2023-03-22 17:56:11,828 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:11,828 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:11,828 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:56:11,828 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,828 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de7176b532804a67b4aa2decef78281c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca51fe62c52b4bd19313415f7e411482, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a01c5b4a1d294d04a9b5073b9fcb53d8] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=36.9 K 2023-03-22 17:56:11,828 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting de7176b532804a67b4aa2decef78281c, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1679507770325 2023-03-22 17:56:11,829 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ca51fe62c52b4bd19313415f7e411482, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1679507770679 2023-03-22 17:56:11,829 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a01c5b4a1d294d04a9b5073b9fcb53d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1679507771317 2023-03-22 17:56:11,834 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#499 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:11,846 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/02468db71b0548ae8b472aa0e415beaa as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/02468db71b0548ae8b472aa0e415beaa 2023-03-22 17:56:11,850 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into 02468db71b0548ae8b472aa0e415beaa(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:11,850 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,850 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=13, startTime=1679507771796; duration=0sec 2023-03-22 17:56:11,850 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:11,873 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:11,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:11,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:11,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:11,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:11,876 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:11,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:11,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:11,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:11,879 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:11,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:11,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:11,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:11,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:11,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:11,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:11,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:11,880 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,880 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:11,881 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-03-22 17:56:11,881 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:11,881 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:11,881 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:11,881 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:11,881 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:11,881 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:11,889 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/bec7c01c0ffd476da50c14fa01b22255 2023-03-22 17:56:11,898 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/c4c106ac0dc149e3b0270266a3ddc667 2023-03-22 17:56:11,917 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/eec824fae6b24969bc15d277c1917d32 2023-03-22 17:56:11,920 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/bec7c01c0ffd476da50c14fa01b22255 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/bec7c01c0ffd476da50c14fa01b22255 2023-03-22 17:56:11,923 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/bec7c01c0ffd476da50c14fa01b22255, entries=150, sequenceid=725, filesize=11.9 K 2023-03-22 17:56:11,924 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/c4c106ac0dc149e3b0270266a3ddc667 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4c106ac0dc149e3b0270266a3ddc667 2023-03-22 17:56:11,928 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4c106ac0dc149e3b0270266a3ddc667, entries=150, sequenceid=725, filesize=11.9 K 2023-03-22 17:56:11,929 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/eec824fae6b24969bc15d277c1917d32 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/eec824fae6b24969bc15d277c1917d32 2023-03-22 17:56:11,932 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/eec824fae6b24969bc15d277c1917d32, entries=150, sequenceid=725, filesize=11.9 K 2023-03-22 17:56:11,933 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=0 B/0 for 523fd4cd7256d4e27c9367c2949e4282 in 52ms, sequenceid=725, compaction requested=false 2023-03-22 17:56:11,933 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:11,933 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:11,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@291124d2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:11,938 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:11,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,940 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,940 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:11,940 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:11,940 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:11,941 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:11,941 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,941 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:11,941 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:11,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:11,942 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:11,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:11,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:11,942 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:11,942 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:11,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:11,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:11,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:11,944 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,946 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,949 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:11,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:11,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:11,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:11,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:11,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:11,950 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:11,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:11,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:11,950 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:11,950 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:11,950 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:11,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:11,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:12,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:12,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:12,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:12,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:12,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:12,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:12,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:12,050 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:12,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:12,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:12,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:12,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:12,054 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:12,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:12,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:12,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,056 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:12,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:12,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:12,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:12,057 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,057 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:12,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:12,057 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:12,057 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:12,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:12,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:12,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:12,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:12,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52d9308a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:12,060 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:12,060 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,062 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:12,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:12,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:12,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,062 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:12,064 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,064 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:12,064 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:12,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,064 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:12,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:12,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:12,068 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:12,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:12,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:12,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:12,068 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:12,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507772046, max timestamp=1679507772067Num uniques 859; 2023-03-22 17:56:12,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:56:12,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507772046, max timestamp=1679507772067Num uniques 859; 2023-03-22 17:56:12,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679507772046, max timestamp=1679507772067Num uniques 859; 2023-03-22 17:56:12,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:56:12,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-22 17:56:12,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:12,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,074 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,075 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,076 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,081 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:12,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:12,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,081 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,081 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:12,081 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:12,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:12,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507832085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507832093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507832094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507832095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507832096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,181 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:12,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:12,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:12,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:12,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:12,185 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:12,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:12,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:12,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,185 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,187 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:12,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:12,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:12,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:12,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:12,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:12,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:12,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:12,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:12,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:12,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:12,191 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,192 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:12,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c86be5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:12,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:12,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,194 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:12,194 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:12,194 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:12,195 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,195 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:12,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:12,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:12,199 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:12,199 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:12,199 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:12,199 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:12,199 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:12,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507832197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507832197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507832200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,205 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,206 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,207 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,210 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:12,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:12,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,210 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,210 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:12,210 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:12,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507832207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507832208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,311 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:12,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:12,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:12,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:12,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:12,314 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:12,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:12,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:12,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,316 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,316 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:12,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:12,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:12,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:12,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:12,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:12,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:12,331 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,332 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:12,332 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:12,332 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,332 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:12,332 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:12,332 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,332 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:12,332 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:12,333 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,333 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,334 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,334 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:12,335 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:12,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d9f9e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:12,336 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,337 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:12,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:12,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:12,338 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,338 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:12,339 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,339 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,339 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:12,339 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:12,341 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:12,341 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:12,341 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:12,341 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:12,342 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:12,343 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,343 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,343 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,344 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,348 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:12,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:12,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:12,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,351 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:12,351 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:12,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507832403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507832404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507832405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507832414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507832414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,444 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/a5ee289c5a1e426ba3685fe923a6a19e 2023-03-22 17:56:12,451 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:12,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:12,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/3850d57d3e824d75ab585d40a07fe0e5 2023-03-22 17:56:12,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:12,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:12,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:12,455 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:12,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:12,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:12,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,458 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,458 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:12,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:12,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:12,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:12,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:12,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:12,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:12,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:12,460 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,460 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:12,460 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:12,460 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:12,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:12,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:12,460 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:12,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:12,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@219a1cd9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:12,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:12,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,471 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:12,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ecc0ea0c84894b61a1f2a0cf829013e3 2023-03-22 17:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:12,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,472 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:12,473 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:12,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,473 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/a5ee289c5a1e426ba3685fe923a6a19e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a5ee289c5a1e426ba3685fe923a6a19e 2023-03-22 17:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:12,477 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:12,477 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:12,477 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:12,477 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:12,477 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:12,480 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,480 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:12,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a5ee289c5a1e426ba3685fe923a6a19e, entries=200, sequenceid=736, filesize=14.3 K 2023-03-22 17:56:12,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:12,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/3850d57d3e824d75ab585d40a07fe0e5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/3850d57d3e824d75ab585d40a07fe0e5 2023-03-22 17:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:12,482 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:12,484 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/3850d57d3e824d75ab585d40a07fe0e5, entries=150, sequenceid=736, filesize=11.9 K 2023-03-22 17:56:12,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ecc0ea0c84894b61a1f2a0cf829013e3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ecc0ea0c84894b61a1f2a0cf829013e3 2023-03-22 17:56:12,489 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:12,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:12,489 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:12,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,489 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:12,489 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:12,490 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:12,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:12,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ecc0ea0c84894b61a1f2a0cf829013e3, entries=150, sequenceid=736, filesize=11.9 K 2023-03-22 17:56:12,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:12,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,492 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 523fd4cd7256d4e27c9367c2949e4282 in 457ms, sequenceid=736, compaction requested=true 2023-03-22 17:56:12,492 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:12,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:12,493 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:12,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:12,493 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:12,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:12,496 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40394 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:12,496 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:12,496 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:56:12,496 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:56:12,496 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,496 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,496 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f781447dc0e34fb5b9921123c593058d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/bec7c01c0ffd476da50c14fa01b22255, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a5ee289c5a1e426ba3685fe923a6a19e] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=39.4 K 2023-03-22 17:56:12,496 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9dd972dea5e64c9195abb6513cecbeb8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4c106ac0dc149e3b0270266a3ddc667, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/3850d57d3e824d75ab585d40a07fe0e5] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=37.1 K 2023-03-22 17:56:12,497 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f781447dc0e34fb5b9921123c593058d, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1679507771317 2023-03-22 17:56:12,497 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9dd972dea5e64c9195abb6513cecbeb8, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1679507771317 2023-03-22 17:56:12,497 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting bec7c01c0ffd476da50c14fa01b22255, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1679507771405 2023-03-22 17:56:12,497 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting c4c106ac0dc149e3b0270266a3ddc667, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1679507771405 2023-03-22 17:56:12,497 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a5ee289c5a1e426ba3685fe923a6a19e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1679507772030 2023-03-22 17:56:12,497 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3850d57d3e824d75ab585d40a07fe0e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1679507772034 2023-03-22 17:56:12,503 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#507 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:12,503 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:12,590 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:12,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:12,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:12,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:12,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:12,593 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:12,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:12,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:12,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,593 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:12,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:12,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,598 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:12,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:12,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:12,599 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:12,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,599 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:12,599 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-03-22 17:56:12,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:12,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:12,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:12,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:12,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:12,600 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:12,610 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/76a34e32d56247299cd445d43c91d353 2023-03-22 17:56:12,625 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/2dc6765c5dd44b3d90f4580c9afc46d7 2023-03-22 17:56:12,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:12,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:12,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507832721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507832723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507832729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507832734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507832734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507832835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507832843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507832843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:12,918 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/dabc9c83470840c99e6cb356e72b20c7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/dabc9c83470840c99e6cb356e72b20c7 2023-03-22 17:56:12,922 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into dabc9c83470840c99e6cb356e72b20c7(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:12,922 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:12,922 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=13, startTime=1679507772493; duration=0sec 2023-03-22 17:56:12,922 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:12,922 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:12,923 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:12,923 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:56:12,924 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:12,924 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/02468db71b0548ae8b472aa0e415beaa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/eec824fae6b24969bc15d277c1917d32, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ecc0ea0c84894b61a1f2a0cf829013e3] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=37.0 K 2023-03-22 17:56:12,924 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 02468db71b0548ae8b472aa0e415beaa, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1679507771317 2023-03-22 17:56:12,924 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting eec824fae6b24969bc15d277c1917d32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1679507771405 2023-03-22 17:56:12,925 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ecc0ea0c84894b61a1f2a0cf829013e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1679507772034 2023-03-22 17:56:12,927 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/827cdcce27fe49558cb3a16d6deb2808 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/827cdcce27fe49558cb3a16d6deb2808 2023-03-22 17:56:12,930 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#511 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:12,931 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into 827cdcce27fe49558cb3a16d6deb2808(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:12,931 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:12,931 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=13, startTime=1679507772492; duration=0sec 2023-03-22 17:56:12,931 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:13,042 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/0b022825f0a3473db898914312eea06b 2023-03-22 17:56:13,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507833041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,046 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/76a34e32d56247299cd445d43c91d353 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/76a34e32d56247299cd445d43c91d353 2023-03-22 17:56:13,050 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/76a34e32d56247299cd445d43c91d353, entries=150, sequenceid=770, filesize=11.9 K 2023-03-22 17:56:13,051 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/2dc6765c5dd44b3d90f4580c9afc46d7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/2dc6765c5dd44b3d90f4580c9afc46d7 2023-03-22 17:56:13,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507833048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,054 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/2dc6765c5dd44b3d90f4580c9afc46d7, entries=150, sequenceid=770, filesize=11.9 K 2023-03-22 17:56:13,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507833049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,055 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/0b022825f0a3473db898914312eea06b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0b022825f0a3473db898914312eea06b 2023-03-22 17:56:13,059 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0b022825f0a3473db898914312eea06b, entries=150, sequenceid=770, filesize=11.9 K 2023-03-22 17:56:13,060 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=60.38 KB/61830 for 523fd4cd7256d4e27c9367c2949e4282 in 461ms, sequenceid=770, compaction requested=false 2023-03-22 17:56:13,060 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,060 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:13,062 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,062 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5c3500[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:13,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:13,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,066 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:13,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:13,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:13,066 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:13,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:13,070 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:13,070 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:13,070 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:13,070 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:13,070 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:13,073 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,073 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,073 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:13,073 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,075 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:13,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:13,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:13,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:13,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:13,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:13,077 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:13,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,077 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,078 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:13,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,078 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:13,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,078 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:13,178 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:13,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:13,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:13,181 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:13,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:13,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:13,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:13,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,183 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:13,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:13,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:13,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:13,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:13,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:13,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:13,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:13,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:13,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:13,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,185 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:13,186 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-22 17:56:13,186 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:13,186 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,186 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:13,186 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,186 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:13,186 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,194 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/edcd0c329a3645d2af06785aec47502d 2023-03-22 17:56:13,208 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/7c01d90b1bed42449b705c9a3a5d9313 2023-03-22 17:56:13,222 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ee741a76246c4f5cabf2eb3d709ec9b4 2023-03-22 17:56:13,226 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/edcd0c329a3645d2af06785aec47502d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/edcd0c329a3645d2af06785aec47502d 2023-03-22 17:56:13,229 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/edcd0c329a3645d2af06785aec47502d, entries=150, sequenceid=784, filesize=11.9 K 2023-03-22 17:56:13,230 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/7c01d90b1bed42449b705c9a3a5d9313 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7c01d90b1bed42449b705c9a3a5d9313 2023-03-22 17:56:13,233 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7c01d90b1bed42449b705c9a3a5d9313, entries=150, sequenceid=784, filesize=11.9 K 2023-03-22 17:56:13,234 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ee741a76246c4f5cabf2eb3d709ec9b4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee741a76246c4f5cabf2eb3d709ec9b4 2023-03-22 17:56:13,238 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee741a76246c4f5cabf2eb3d709ec9b4, entries=150, sequenceid=784, filesize=11.9 K 2023-03-22 17:56:13,239 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 523fd4cd7256d4e27c9367c2949e4282 in 54ms, sequenceid=784, compaction requested=true 2023-03-22 17:56:13,239 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,239 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:13,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:13,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c9e6ed[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:13,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:13,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,246 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:13,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,246 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:13,248 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:13,250 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:13,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:13,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:13,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:13,250 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:13,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:13,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,254 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:13,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,255 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:13,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:13,255 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:13,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:13,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:13,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,257 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:13,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:13,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:13,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:13,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:13,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:13,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,262 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:13,262 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:13,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,262 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:13,263 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/b6e4c6d565b84a4885bfb6f776d42dae 2023-03-22 17:56:13,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/443ef99064764066ba74cefe480ded80 2023-03-22 17:56:13,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/94c84b4ca6b04b6aa11c9c879b255117 2023-03-22 17:56:13,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/b6e4c6d565b84a4885bfb6f776d42dae as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b6e4c6d565b84a4885bfb6f776d42dae 2023-03-22 17:56:13,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b6e4c6d565b84a4885bfb6f776d42dae, entries=250, sequenceid=795, filesize=16.7 K 2023-03-22 17:56:13,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/443ef99064764066ba74cefe480ded80 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/443ef99064764066ba74cefe480ded80 2023-03-22 17:56:13,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/443ef99064764066ba74cefe480ded80, entries=150, sequenceid=795, filesize=11.9 K 2023-03-22 17:56:13,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/94c84b4ca6b04b6aa11c9c879b255117 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/94c84b4ca6b04b6aa11c9c879b255117 2023-03-22 17:56:13,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/94c84b4ca6b04b6aa11c9c879b255117, entries=150, sequenceid=795, filesize=11.9 K 2023-03-22 17:56:13,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=73.80 KB/75570 for 523fd4cd7256d4e27c9367c2949e4282 in 52ms, sequenceid=795, compaction requested=true 2023-03-22 17:56:13,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,307 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:13,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:13,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:13,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-22 17:56:13,307 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55122 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:13,308 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/A is initiating minor compaction (all files) 2023-03-22 17:56:13,308 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/A in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,308 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/827cdcce27fe49558cb3a16d6deb2808, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/76a34e32d56247299cd445d43c91d353, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/edcd0c329a3645d2af06785aec47502d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b6e4c6d565b84a4885bfb6f776d42dae] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=53.8 K 2023-03-22 17:56:13,308 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 827cdcce27fe49558cb3a16d6deb2808, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1679507772034 2023-03-22 17:56:13,308 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 76a34e32d56247299cd445d43c91d353, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1679507772073 2023-03-22 17:56:13,309 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting edcd0c329a3645d2af06785aec47502d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1679507772727 2023-03-22 17:56:13,309 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b6e4c6d565b84a4885bfb6f776d42dae, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1679507773246 2023-03-22 17:56:13,315 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#A#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:13,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:13,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-22 17:56:13,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:13,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:13,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:13,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:13,325 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/c816436145304850912777e0ca5ea514 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c816436145304850912777e0ca5ea514 2023-03-22 17:56:13,329 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/A of 523fd4cd7256d4e27c9367c2949e4282 into c816436145304850912777e0ca5ea514(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:13,329 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,329 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/A, priority=12, startTime=1679507773306; duration=0sec 2023-03-22 17:56:13,329 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-22 17:56:13,329 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 3 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:13,331 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36558 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:13,331 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction 2023-03-22 17:56:13,331 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,331 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0b022825f0a3473db898914312eea06b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee741a76246c4f5cabf2eb3d709ec9b4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/94c84b4ca6b04b6aa11c9c879b255117] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=35.7 K 2023-03-22 17:56:13,332 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0b022825f0a3473db898914312eea06b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770 2023-03-22 17:56:13,332 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting ee741a76246c4f5cabf2eb3d709ec9b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=784 2023-03-22 17:56:13,332 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 94c84b4ca6b04b6aa11c9c879b255117, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795 2023-03-22 17:56:13,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/9185ebe2495b46e6a2cf421097ea5d89 2023-03-22 17:56:13,341 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#520 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:13,345 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/961a5e9ab5c6482f8470d55dace78dee as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/961a5e9ab5c6482f8470d55dace78dee 2023-03-22 17:56:13,366 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:13,366 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/d757304f8c4c4bf8843b9373d32d771a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d757304f8c4c4bf8843b9373d32d771a 2023-03-22 17:56:13,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:13,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:13,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:13,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:13,372 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:13,372 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:13,372 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:13,372 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:13,372 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,372 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,375 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:13,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:13,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:13,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:13,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:13,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:13,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:13,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:13,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:13,380 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,380 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:13,380 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:13,380 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:13,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:13,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:13,380 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,382 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55557f05[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:13,383 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:13,383 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:13,385 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:13,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:13,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:13,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:13,388 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:13,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:13,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:13,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:13,388 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:13,390 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,390 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:13,390 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,391 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,392 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,394 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:13,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:13,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:13,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,394 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:13,394 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:13,394 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:13,398 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into 961a5e9ab5c6482f8470d55dace78dee(size=13.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:13,398 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into d757304f8c4c4bf8843b9373d32d771a(size=12.0 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:13,398 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,398 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,398 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=10, startTime=1679507773307; duration=0sec 2023-03-22 17:56:13,398 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=13, startTime=1679507772493; duration=0sec 2023-03-22 17:56:13,398 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:13,398 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:13,398 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:13,400 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50242 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:13,400 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/B is initiating minor compaction (all files) 2023-03-22 17:56:13,400 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/B in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,400 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/dabc9c83470840c99e6cb356e72b20c7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/2dc6765c5dd44b3d90f4580c9afc46d7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7c01d90b1bed42449b705c9a3a5d9313, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/443ef99064764066ba74cefe480ded80] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=49.1 K 2023-03-22 17:56:13,400 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting dabc9c83470840c99e6cb356e72b20c7, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1679507772034 2023-03-22 17:56:13,401 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 2dc6765c5dd44b3d90f4580c9afc46d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1679507772073 2023-03-22 17:56:13,401 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 7c01d90b1bed42449b705c9a3a5d9313, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1679507772727 2023-03-22 17:56:13,401 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 443ef99064764066ba74cefe480ded80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1679507773251 2023-03-22 17:56:13,410 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#B#compaction#522 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:13,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507833412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507833414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,423 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/45ee5a0c558b48eea10e0af2e58004d0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/45ee5a0c558b48eea10e0af2e58004d0 2023-03-22 17:56:13,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507833414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507833417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507833418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,429 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/B of 523fd4cd7256d4e27c9367c2949e4282 into 45ee5a0c558b48eea10e0af2e58004d0(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:13,429 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,429 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/B, priority=12, startTime=1679507773307; duration=0sec 2023-03-22 17:56:13,429 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:13,495 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:13,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:13,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:13,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:13,499 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:13,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:13,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:13,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:13,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,500 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:13,500 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:13,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,501 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:13,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. started... 2023-03-22 17:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. as already flushing 2023-03-22 17:56:13,502 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:13,503 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:13,507 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,507 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2458dda2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:13,509 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:13,509 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:13,510 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,510 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:13,512 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:13,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:13,514 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:13,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:13,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:13,515 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:13,515 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:13,516 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,516 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:13,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,516 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:13,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:13,517 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,519 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:13,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,523 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:13,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:13,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,524 DEBUG [Thread-3062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:13,524 DEBUG [Thread-3062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:13,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:13,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:13,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:13,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:13,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507833522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507833522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507833526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507833529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507833529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,578 DEBUG [Thread-3069] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:55285 2023-03-22 17:56:13,578 DEBUG [Thread-3069] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:13,580 DEBUG [Thread-3063] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:55285 2023-03-22 17:56:13,580 DEBUG [Thread-3067] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:55285 2023-03-22 17:56:13,580 DEBUG [Thread-3063] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:13,580 DEBUG [Thread-3067] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:13,584 DEBUG [Thread-3071] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:55285 2023-03-22 17:56:13,584 DEBUG [Thread-3071] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:13,585 DEBUG [Thread-3065] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:55285 2023-03-22 17:56:13,586 DEBUG [Thread-3065] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:13,624 DEBUG [Thread-3062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:13,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:13,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54954 deadline: 1679507833726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54980 deadline: 1679507833727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54970 deadline: 1679507833734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:13,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55012 deadline: 1679507833736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54976 deadline: 1679507833736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:13,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/aacfde35eb2740bf897fc92e8c53c11a 2023-03-22 17:56:13,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ee5aeb7a9ccc4aa284dddf4cac648a96 2023-03-22 17:56:13,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/9185ebe2495b46e6a2cf421097ea5d89 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9185ebe2495b46e6a2cf421097ea5d89 2023-03-22 17:56:13,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9185ebe2495b46e6a2cf421097ea5d89, entries=200, sequenceid=810, filesize=14.3 K 2023-03-22 17:56:13,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/aacfde35eb2740bf897fc92e8c53c11a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/aacfde35eb2740bf897fc92e8c53c11a 2023-03-22 17:56:13,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/aacfde35eb2740bf897fc92e8c53c11a, entries=150, sequenceid=810, filesize=11.9 K 2023-03-22 17:56:13,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ee5aeb7a9ccc4aa284dddf4cac648a96 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee5aeb7a9ccc4aa284dddf4cac648a96 2023-03-22 17:56:13,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee5aeb7a9ccc4aa284dddf4cac648a96, entries=150, sequenceid=810, filesize=11.9 K 2023-03-22 17:56:13,803 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 523fd4cd7256d4e27c9367c2949e4282 in 483ms, sequenceid=810, compaction requested=true 2023-03-22 17:56:13,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:13,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-22 17:56:13,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. because compaction request was cancelled 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. because compaction request was cancelled 2023-03-22 17:56:13,804 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:13,805 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38125 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:13,805 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 523fd4cd7256d4e27c9367c2949e4282/C is initiating minor compaction (all files) 2023-03-22 17:56:13,805 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 523fd4cd7256d4e27c9367c2949e4282/C in TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:13,805 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/961a5e9ab5c6482f8470d55dace78dee, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d757304f8c4c4bf8843b9373d32d771a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee5aeb7a9ccc4aa284dddf4cac648a96] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp, totalSize=37.2 K 2023-03-22 17:56:13,805 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 961a5e9ab5c6482f8470d55dace78dee, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1679507772034 2023-03-22 17:56:13,806 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d757304f8c4c4bf8843b9373d32d771a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1679507773251 2023-03-22 17:56:13,806 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting ee5aeb7a9ccc4aa284dddf4cac648a96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=810, earliestPutTs=1679507773286 2023-03-22 17:56:13,810 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 523fd4cd7256d4e27c9367c2949e4282#C#compaction#524 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:13,816 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/ccf80fc99ccc4269a7e329a161081145 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ccf80fc99ccc4269a7e329a161081145 2023-03-22 17:56:13,819 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 523fd4cd7256d4e27c9367c2949e4282/C of 523fd4cd7256d4e27c9367c2949e4282 into ccf80fc99ccc4269a7e329a161081145(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:13,819 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:13,819 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282., storeName=523fd4cd7256d4e27c9367c2949e4282/C, priority=13, startTime=1679507773804; duration=0sec 2023-03-22 17:56:13,819 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:14,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:14,030 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-22 17:56:14,030 DEBUG [Thread-3052] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e372785 to 127.0.0.1:55285 2023-03-22 17:56:14,030 DEBUG [Thread-3052] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:14,030 DEBUG [Thread-3056] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f2d997 to 127.0.0.1:55285 2023-03-22 17:56:14,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:14,030 DEBUG [Thread-3056] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:14,031 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:14,031 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:14,031 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:14,031 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:14,031 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:14,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/82dcba2cfcfd4d05b8c91f7b7633ed7b 2023-03-22 17:56:14,037 DEBUG [Thread-3054] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x501c5a88 to 127.0.0.1:55285 2023-03-22 17:56:14,037 DEBUG [Thread-3054] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:14,038 DEBUG [Thread-3060] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:55285 2023-03-22 17:56:14,038 DEBUG [Thread-3060] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:14,040 DEBUG [Thread-3058] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:55285 2023-03-22 17:56:14,040 DEBUG [Thread-3058] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 180 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 128 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 123 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1666 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 4998 rows 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1673 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 5019 rows 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1675 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 5024 rows 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1674 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 5010 rows 2023-03-22 17:56:14,040 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(402): scanned 1660 2023-03-22 17:56:14,041 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(403): verified 4977 rows 2023-03-22 17:56:14,041 INFO [Listener at localhost/44491] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-22 17:56:14,041 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a29ce5a to 127.0.0.1:55285 2023-03-22 17:56:14,041 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:14,042 INFO [Listener at localhost/44491] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-22 17:56:14,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-22 17:56:14,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:14,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-22 17:56:14,047 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507774047"}]},"ts":"1679507774047"} 2023-03-22 17:56:14,048 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-22 17:56:14,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-22 17:56:14,150 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-22 17:56:14,151 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=523fd4cd7256d4e27c9367c2949e4282, UNASSIGN}] 2023-03-22 17:56:14,152 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=523fd4cd7256d4e27c9367c2949e4282, UNASSIGN 2023-03-22 17:56:14,153 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=523fd4cd7256d4e27c9367c2949e4282, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:14,153 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507774153"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507774153"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507774153"}]},"ts":"1679507774153"} 2023-03-22 17:56:14,154 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:56:14,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:14,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 523fd4cd7256d4e27c9367c2949e4282, disabling compactions & flushes 2023-03-22 17:56:14,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:14,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-22 17:56:14,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/f1487968a4d44c89bdea95e55ef1804a 2023-03-22 17:56:14,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-22 17:56:14,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/0c1a680480d74a91a9920e951e20faea 2023-03-22 17:56:14,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/82dcba2cfcfd4d05b8c91f7b7633ed7b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/82dcba2cfcfd4d05b8c91f7b7633ed7b 2023-03-22 17:56:14,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/82dcba2cfcfd4d05b8c91f7b7633ed7b, entries=150, sequenceid=834, filesize=11.9 K 2023-03-22 17:56:14,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/f1487968a4d44c89bdea95e55ef1804a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f1487968a4d44c89bdea95e55ef1804a 2023-03-22 17:56:14,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f1487968a4d44c89bdea95e55ef1804a, entries=150, sequenceid=834, filesize=11.9 K 2023-03-22 17:56:14,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/0c1a680480d74a91a9920e951e20faea as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c1a680480d74a91a9920e951e20faea 2023-03-22 17:56:14,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c1a680480d74a91a9920e951e20faea, entries=150, sequenceid=834, filesize=11.9 K 2023-03-22 17:56:14,873 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=20.13 KB/20610 for 523fd4cd7256d4e27c9367c2949e4282 in 843ms, sequenceid=834, compaction requested=true 2023-03-22 17:56:14,873 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:14,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:14,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:14,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:14,873 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. because compaction request was cancelled 2023-03-22 17:56:14,873 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. because compaction request was cancelled 2023-03-22 17:56:14,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. after waiting 0 ms 2023-03-22 17:56:14,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:14,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:14,873 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. because compaction request was cancelled 2023-03-22 17:56:14,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:14,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 523fd4cd7256d4e27c9367c2949e4282 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-22 17:56:14,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=A 2023-03-22 17:56:14,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:14,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=B 2023-03-22 17:56:14,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:14,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 523fd4cd7256d4e27c9367c2949e4282, store=C 2023-03-22 17:56:14,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:14,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/1eb1ebb31e2443d19b9c85959648fbed 2023-03-22 17:56:15,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-22 17:56:15,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/40526d8a8c7544129f41d6782d93d518 2023-03-22 17:56:15,694 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/4858af5a4a0341fcb62d4411ab8b96bf 2023-03-22 17:56:15,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/A/1eb1ebb31e2443d19b9c85959648fbed as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1eb1ebb31e2443d19b9c85959648fbed 2023-03-22 17:56:15,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1eb1ebb31e2443d19b9c85959648fbed, entries=100, sequenceid=840, filesize=9.5 K 2023-03-22 17:56:15,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/B/40526d8a8c7544129f41d6782d93d518 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/40526d8a8c7544129f41d6782d93d518 2023-03-22 17:56:15,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/40526d8a8c7544129f41d6782d93d518, entries=100, sequenceid=840, filesize=9.5 K 2023-03-22 17:56:15,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/.tmp/C/4858af5a4a0341fcb62d4411ab8b96bf as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/4858af5a4a0341fcb62d4411ab8b96bf 2023-03-22 17:56:15,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/4858af5a4a0341fcb62d4411ab8b96bf, entries=100, sequenceid=840, filesize=9.5 K 2023-03-22 17:56:15,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 523fd4cd7256d4e27c9367c2949e4282 in 834ms, sequenceid=840, compaction requested=true 2023-03-22 17:56:15,708 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/4fdbfd30ea0544cb8b10c194edfb113f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fb638a6cf004d77acb03f9102d1bef9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/32f266bed58744b88ff9b073554e14b6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/cb1cb9f2ebf64154865620cc13d4302f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/453528ac90d7472995fcdeb6183e74c9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b9ace1850c7d4e0dbf1036aea1bc72a8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1b98d5acb4ed40d1a7795659aeb46c99, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/0cdc37ce18154e18999999e956cd2a1c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/31ff69c0fa014c619b49155da97a2364, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/271515a76d984653a7643ff150cb2ca0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fd8dc3de1ee4aada330b48850cef896, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c71f28e67a2f48bd94cb20fc2f9d5247, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f6f1ad4de271469ebd35414282d0f3cd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/552b544636cb44518cccc2c9fca9ff48, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b4ff3eea01db482e8086092dd9d52411, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/138c50bf60354ddf83d1962b4dea51d0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9145089327f449d0be732c8ab5d9ccaa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1e6b7f5e51484f3990ca72124568ecd9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/083f4c71369d426198c6d2cd580fe773, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/123ecd74bf4345a08afd4d2494013493, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a6cb260f3ff8465f919aa7c846a2fe3b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/347496a40f434e76bde7238b13861927, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/e9bd3a9074dc482d986b19a70fd5d758, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/7f56a5ded9394cac965dc49fb9c0076c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/85c94bff4e8b489482f394da09b6493b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c19e440daae64832989d0ce83d926264, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/6be6749fe840417285fc9ee33c2687a6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d8d4ed351b834320b5b8784dcceae0a3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/ab3c4c9763c24ba484ce52ba97e19ace, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d71f170926d441e7b219e7bd62dc1cf9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fe951b11ed2f4f9d9743fd7dc614529a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fbffa8127bfd45f9bc6413bc0311e79c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a22ba1a151304b2a8b25118637bfc571, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9c823d4e5b3a4be2ade6497b9e210696, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/64378d9d76f34b3aacf4820b32091b8a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/19fcf2e3d3c64abe99255677bd5f32e4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/913b665a6a824e4d871b5971e46f203a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/3b34c7f2e4f040918f6e41514bc91818, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d22e5d29b3c74c42b28da0a5b0d62070, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d851a24d43ec408e9d75af72f47a5ced, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f781447dc0e34fb5b9921123c593058d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a9df7b58eebf41439d992a089eb568c3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/bec7c01c0ffd476da50c14fa01b22255, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a5ee289c5a1e426ba3685fe923a6a19e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/827cdcce27fe49558cb3a16d6deb2808, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/76a34e32d56247299cd445d43c91d353, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/edcd0c329a3645d2af06785aec47502d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b6e4c6d565b84a4885bfb6f776d42dae] to archive 2023-03-22 17:56:15,709 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:56:15,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/4fdbfd30ea0544cb8b10c194edfb113f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/4fdbfd30ea0544cb8b10c194edfb113f 2023-03-22 17:56:15,712 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fb638a6cf004d77acb03f9102d1bef9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fb638a6cf004d77acb03f9102d1bef9 2023-03-22 17:56:15,713 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/32f266bed58744b88ff9b073554e14b6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/32f266bed58744b88ff9b073554e14b6 2023-03-22 17:56:15,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/cb1cb9f2ebf64154865620cc13d4302f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/cb1cb9f2ebf64154865620cc13d4302f 2023-03-22 17:56:15,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/453528ac90d7472995fcdeb6183e74c9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/453528ac90d7472995fcdeb6183e74c9 2023-03-22 17:56:15,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b9ace1850c7d4e0dbf1036aea1bc72a8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b9ace1850c7d4e0dbf1036aea1bc72a8 2023-03-22 17:56:15,716 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1b98d5acb4ed40d1a7795659aeb46c99 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1b98d5acb4ed40d1a7795659aeb46c99 2023-03-22 17:56:15,717 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/0cdc37ce18154e18999999e956cd2a1c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/0cdc37ce18154e18999999e956cd2a1c 2023-03-22 17:56:15,718 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/31ff69c0fa014c619b49155da97a2364 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/31ff69c0fa014c619b49155da97a2364 2023-03-22 17:56:15,718 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/271515a76d984653a7643ff150cb2ca0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/271515a76d984653a7643ff150cb2ca0 2023-03-22 17:56:15,719 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fd8dc3de1ee4aada330b48850cef896 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/8fd8dc3de1ee4aada330b48850cef896 2023-03-22 17:56:15,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c71f28e67a2f48bd94cb20fc2f9d5247 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c71f28e67a2f48bd94cb20fc2f9d5247 2023-03-22 17:56:15,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f6f1ad4de271469ebd35414282d0f3cd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f6f1ad4de271469ebd35414282d0f3cd 2023-03-22 17:56:15,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/552b544636cb44518cccc2c9fca9ff48 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/552b544636cb44518cccc2c9fca9ff48 2023-03-22 17:56:15,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b4ff3eea01db482e8086092dd9d52411 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b4ff3eea01db482e8086092dd9d52411 2023-03-22 17:56:15,723 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/138c50bf60354ddf83d1962b4dea51d0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/138c50bf60354ddf83d1962b4dea51d0 2023-03-22 17:56:15,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9145089327f449d0be732c8ab5d9ccaa to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9145089327f449d0be732c8ab5d9ccaa 2023-03-22 17:56:15,725 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1e6b7f5e51484f3990ca72124568ecd9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1e6b7f5e51484f3990ca72124568ecd9 2023-03-22 17:56:15,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/083f4c71369d426198c6d2cd580fe773 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/083f4c71369d426198c6d2cd580fe773 2023-03-22 17:56:15,729 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/123ecd74bf4345a08afd4d2494013493 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/123ecd74bf4345a08afd4d2494013493 2023-03-22 17:56:15,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a6cb260f3ff8465f919aa7c846a2fe3b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a6cb260f3ff8465f919aa7c846a2fe3b 2023-03-22 17:56:15,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/347496a40f434e76bde7238b13861927 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/347496a40f434e76bde7238b13861927 2023-03-22 17:56:15,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/e9bd3a9074dc482d986b19a70fd5d758 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/e9bd3a9074dc482d986b19a70fd5d758 2023-03-22 17:56:15,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/7f56a5ded9394cac965dc49fb9c0076c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/7f56a5ded9394cac965dc49fb9c0076c 2023-03-22 17:56:15,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/85c94bff4e8b489482f394da09b6493b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/85c94bff4e8b489482f394da09b6493b 2023-03-22 17:56:15,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c19e440daae64832989d0ce83d926264 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c19e440daae64832989d0ce83d926264 2023-03-22 17:56:15,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/6be6749fe840417285fc9ee33c2687a6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/6be6749fe840417285fc9ee33c2687a6 2023-03-22 17:56:15,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d8d4ed351b834320b5b8784dcceae0a3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d8d4ed351b834320b5b8784dcceae0a3 2023-03-22 17:56:15,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/ab3c4c9763c24ba484ce52ba97e19ace to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/ab3c4c9763c24ba484ce52ba97e19ace 2023-03-22 17:56:15,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d71f170926d441e7b219e7bd62dc1cf9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d71f170926d441e7b219e7bd62dc1cf9 2023-03-22 17:56:15,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fe951b11ed2f4f9d9743fd7dc614529a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fe951b11ed2f4f9d9743fd7dc614529a 2023-03-22 17:56:15,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fbffa8127bfd45f9bc6413bc0311e79c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/fbffa8127bfd45f9bc6413bc0311e79c 2023-03-22 17:56:15,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a22ba1a151304b2a8b25118637bfc571 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a22ba1a151304b2a8b25118637bfc571 2023-03-22 17:56:15,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9c823d4e5b3a4be2ade6497b9e210696 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9c823d4e5b3a4be2ade6497b9e210696 2023-03-22 17:56:15,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/64378d9d76f34b3aacf4820b32091b8a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/64378d9d76f34b3aacf4820b32091b8a 2023-03-22 17:56:15,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/19fcf2e3d3c64abe99255677bd5f32e4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/19fcf2e3d3c64abe99255677bd5f32e4 2023-03-22 17:56:15,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/913b665a6a824e4d871b5971e46f203a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/913b665a6a824e4d871b5971e46f203a 2023-03-22 17:56:15,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/3b34c7f2e4f040918f6e41514bc91818 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/3b34c7f2e4f040918f6e41514bc91818 2023-03-22 17:56:15,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d22e5d29b3c74c42b28da0a5b0d62070 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d22e5d29b3c74c42b28da0a5b0d62070 2023-03-22 17:56:15,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d851a24d43ec408e9d75af72f47a5ced to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/d851a24d43ec408e9d75af72f47a5ced 2023-03-22 17:56:15,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f781447dc0e34fb5b9921123c593058d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/f781447dc0e34fb5b9921123c593058d 2023-03-22 17:56:15,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a9df7b58eebf41439d992a089eb568c3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a9df7b58eebf41439d992a089eb568c3 2023-03-22 17:56:15,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/bec7c01c0ffd476da50c14fa01b22255 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/bec7c01c0ffd476da50c14fa01b22255 2023-03-22 17:56:15,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a5ee289c5a1e426ba3685fe923a6a19e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/a5ee289c5a1e426ba3685fe923a6a19e 2023-03-22 17:56:15,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/827cdcce27fe49558cb3a16d6deb2808 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/827cdcce27fe49558cb3a16d6deb2808 2023-03-22 17:56:15,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/76a34e32d56247299cd445d43c91d353 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/76a34e32d56247299cd445d43c91d353 2023-03-22 17:56:15,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/edcd0c329a3645d2af06785aec47502d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/edcd0c329a3645d2af06785aec47502d 2023-03-22 17:56:15,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b6e4c6d565b84a4885bfb6f776d42dae to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/b6e4c6d565b84a4885bfb6f776d42dae 2023-03-22 17:56:15,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/b28bc1bf2f444391a0373c17b010aaa0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/52cbe939246e47948e69113de2b35c02, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/94f3a541d4fc46d6b10d81a82d9c488f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0795b7a5d39e42b2a4cacc02c19cc0de, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/38fce18265604c64a8c4efc3c467890b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fd9ab0d76d334cde9866f6b19822d9dc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/8b9b0a588399477caf34e478e69019e0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4aa12ec770b4b339aa2028ab4cdab07, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f4262ea2347e4dc29f6360ff2ffea51d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fbe18bee0a1746cb8f1eb37a9d5e0ad4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7d04e14997ac49969c02db82c4ce0daa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9fe2410beea54d86b6c812efc5b584e6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/63a5de226aef46089330ffc901b905db, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/033f384611f645749f56cc0e097486ec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/edf921d2c0724d999982ab8a6f4763ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f13c0d350a174c20bac24a526d179960, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1c8e67a3ad964609a7ef6301fec74641, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/bf3797b34c6f486fb1c0130f5690f9a9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/20afb051e550486fa29b6f542e91473f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6bd18fb515754a39801403c9955d2ef6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5b3e7ed920214dc095b4670ba43c0fe2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5e8c79d4924d4dbcb08474ac3c53a070, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/de3238e6a6584cf58b45cd7a72c3b9aa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/4f83eb768c8d45b99728534dc94cdbd0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0f22b662b6974d5c98df9aa7feff7f3a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fc6fa3aee12443e080b816cdcf724701, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/27472a70c6ed491b943bd5795c5e28a7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/878948773ad8418fa1e4f86f5ba0a666, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/cda2c8d84c3742dcb2848439bebd2772, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0ed553bebe7647b9b1c40b7c8ffb5b13, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9b8eba02f31445239341ac39cb9b5534, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6a98b00d68fa459cb59938ebf1fe7cc3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/62fd9ed54dd0443fab32087f343e41b4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/e2fe5cd8b3534691b7a48e1021b9d52b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6527d8b7b6fa40f0a598681d55aa249d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/214e1c6374004adaae1fddf5e0d055e7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/d8c897d22a8747f6bce1615ad7e9d141, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/ce0575d00a4e49f2a8ad8e6ca9be3571, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f06fa8d3e2b54493b351c91dbc166e99, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/19252d7d8d084a6da728d26f7c107483, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9dd972dea5e64c9195abb6513cecbeb8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1bed71c9261f4021aaef7b6d4b732029, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4c106ac0dc149e3b0270266a3ddc667, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/dabc9c83470840c99e6cb356e72b20c7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/3850d57d3e824d75ab585d40a07fe0e5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/2dc6765c5dd44b3d90f4580c9afc46d7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7c01d90b1bed42449b705c9a3a5d9313, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/443ef99064764066ba74cefe480ded80] to archive 2023-03-22 17:56:15,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:56:15,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/b28bc1bf2f444391a0373c17b010aaa0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/b28bc1bf2f444391a0373c17b010aaa0 2023-03-22 17:56:15,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/52cbe939246e47948e69113de2b35c02 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/52cbe939246e47948e69113de2b35c02 2023-03-22 17:56:15,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/94f3a541d4fc46d6b10d81a82d9c488f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/94f3a541d4fc46d6b10d81a82d9c488f 2023-03-22 17:56:15,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0795b7a5d39e42b2a4cacc02c19cc0de to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0795b7a5d39e42b2a4cacc02c19cc0de 2023-03-22 17:56:15,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/38fce18265604c64a8c4efc3c467890b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/38fce18265604c64a8c4efc3c467890b 2023-03-22 17:56:15,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fd9ab0d76d334cde9866f6b19822d9dc to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fd9ab0d76d334cde9866f6b19822d9dc 2023-03-22 17:56:15,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/8b9b0a588399477caf34e478e69019e0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/8b9b0a588399477caf34e478e69019e0 2023-03-22 17:56:15,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4aa12ec770b4b339aa2028ab4cdab07 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4aa12ec770b4b339aa2028ab4cdab07 2023-03-22 17:56:15,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f4262ea2347e4dc29f6360ff2ffea51d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f4262ea2347e4dc29f6360ff2ffea51d 2023-03-22 17:56:15,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fbe18bee0a1746cb8f1eb37a9d5e0ad4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fbe18bee0a1746cb8f1eb37a9d5e0ad4 2023-03-22 17:56:15,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7d04e14997ac49969c02db82c4ce0daa to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7d04e14997ac49969c02db82c4ce0daa 2023-03-22 17:56:15,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9fe2410beea54d86b6c812efc5b584e6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9fe2410beea54d86b6c812efc5b584e6 2023-03-22 17:56:15,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/63a5de226aef46089330ffc901b905db to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/63a5de226aef46089330ffc901b905db 2023-03-22 17:56:15,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/033f384611f645749f56cc0e097486ec to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/033f384611f645749f56cc0e097486ec 2023-03-22 17:56:15,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/edf921d2c0724d999982ab8a6f4763ae to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/edf921d2c0724d999982ab8a6f4763ae 2023-03-22 17:56:15,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f13c0d350a174c20bac24a526d179960 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f13c0d350a174c20bac24a526d179960 2023-03-22 17:56:15,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1c8e67a3ad964609a7ef6301fec74641 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1c8e67a3ad964609a7ef6301fec74641 2023-03-22 17:56:15,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/bf3797b34c6f486fb1c0130f5690f9a9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/bf3797b34c6f486fb1c0130f5690f9a9 2023-03-22 17:56:15,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/20afb051e550486fa29b6f542e91473f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/20afb051e550486fa29b6f542e91473f 2023-03-22 17:56:15,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6bd18fb515754a39801403c9955d2ef6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6bd18fb515754a39801403c9955d2ef6 2023-03-22 17:56:15,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5b3e7ed920214dc095b4670ba43c0fe2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5b3e7ed920214dc095b4670ba43c0fe2 2023-03-22 17:56:15,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5e8c79d4924d4dbcb08474ac3c53a070 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/5e8c79d4924d4dbcb08474ac3c53a070 2023-03-22 17:56:15,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/de3238e6a6584cf58b45cd7a72c3b9aa to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/de3238e6a6584cf58b45cd7a72c3b9aa 2023-03-22 17:56:15,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/4f83eb768c8d45b99728534dc94cdbd0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/4f83eb768c8d45b99728534dc94cdbd0 2023-03-22 17:56:15,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0f22b662b6974d5c98df9aa7feff7f3a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0f22b662b6974d5c98df9aa7feff7f3a 2023-03-22 17:56:15,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fc6fa3aee12443e080b816cdcf724701 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/fc6fa3aee12443e080b816cdcf724701 2023-03-22 17:56:15,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/27472a70c6ed491b943bd5795c5e28a7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/27472a70c6ed491b943bd5795c5e28a7 2023-03-22 17:56:15,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/878948773ad8418fa1e4f86f5ba0a666 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/878948773ad8418fa1e4f86f5ba0a666 2023-03-22 17:56:15,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/cda2c8d84c3742dcb2848439bebd2772 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/cda2c8d84c3742dcb2848439bebd2772 2023-03-22 17:56:15,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0ed553bebe7647b9b1c40b7c8ffb5b13 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/0ed553bebe7647b9b1c40b7c8ffb5b13 2023-03-22 17:56:15,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9b8eba02f31445239341ac39cb9b5534 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9b8eba02f31445239341ac39cb9b5534 2023-03-22 17:56:15,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6a98b00d68fa459cb59938ebf1fe7cc3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6a98b00d68fa459cb59938ebf1fe7cc3 2023-03-22 17:56:15,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/62fd9ed54dd0443fab32087f343e41b4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/62fd9ed54dd0443fab32087f343e41b4 2023-03-22 17:56:15,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/e2fe5cd8b3534691b7a48e1021b9d52b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/e2fe5cd8b3534691b7a48e1021b9d52b 2023-03-22 17:56:15,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6527d8b7b6fa40f0a598681d55aa249d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/6527d8b7b6fa40f0a598681d55aa249d 2023-03-22 17:56:15,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/214e1c6374004adaae1fddf5e0d055e7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/214e1c6374004adaae1fddf5e0d055e7 2023-03-22 17:56:15,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/d8c897d22a8747f6bce1615ad7e9d141 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/d8c897d22a8747f6bce1615ad7e9d141 2023-03-22 17:56:15,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/ce0575d00a4e49f2a8ad8e6ca9be3571 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/ce0575d00a4e49f2a8ad8e6ca9be3571 2023-03-22 17:56:15,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f06fa8d3e2b54493b351c91dbc166e99 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f06fa8d3e2b54493b351c91dbc166e99 2023-03-22 17:56:15,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/19252d7d8d084a6da728d26f7c107483 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/19252d7d8d084a6da728d26f7c107483 2023-03-22 17:56:15,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9dd972dea5e64c9195abb6513cecbeb8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/9dd972dea5e64c9195abb6513cecbeb8 2023-03-22 17:56:15,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1bed71c9261f4021aaef7b6d4b732029 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/1bed71c9261f4021aaef7b6d4b732029 2023-03-22 17:56:15,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4c106ac0dc149e3b0270266a3ddc667 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/c4c106ac0dc149e3b0270266a3ddc667 2023-03-22 17:56:15,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/dabc9c83470840c99e6cb356e72b20c7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/dabc9c83470840c99e6cb356e72b20c7 2023-03-22 17:56:15,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/3850d57d3e824d75ab585d40a07fe0e5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/3850d57d3e824d75ab585d40a07fe0e5 2023-03-22 17:56:15,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/2dc6765c5dd44b3d90f4580c9afc46d7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/2dc6765c5dd44b3d90f4580c9afc46d7 2023-03-22 17:56:15,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7c01d90b1bed42449b705c9a3a5d9313 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/7c01d90b1bed42449b705c9a3a5d9313 2023-03-22 17:56:15,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/443ef99064764066ba74cefe480ded80 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/443ef99064764066ba74cefe480ded80 2023-03-22 17:56:15,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c4a1d79cce1d465792063315973694f5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5ca93ffdaa914f83a60afdb71f66ad9c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/6d034e6aabad47958089f784b4fb7b6c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb9b0c4f73e445819a51a8618c4325ab, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d5e6586101754eaabec80e2e8b2b209a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca6c77931d5b45d2920917c762ea5779, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/320b138d0af3499288c641226014f2ec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/540b46a9293e4bf19ef77170e724462c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/09c320cd9f8e46d08d93aa5999f62657, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/fdde32930d0a443ca358fd7e04548df8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e443418e7b01422c848983ab896e10fa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a603ee386afa4c6cba9878c2d01acc2c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d7c78d749c6843b6874c6e0ec477ee6d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5e52e6f3806747698cddb5183d35ec55, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d8e3d770c26c474882e549d8937bd57d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e9a5ebcadd7b4c92b1bd8a77ab15000a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f477e5d2c82c4ee185ab054fe73cd00c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/1eb4b5c4d1ea42d88fb4afcdbabc04f4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c71a52dbe6b84a2f8055ae212130b646, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/651aa0e5fb50496c8b30295eeefaaefb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/9a5615ea28834320a63b10b1d27ca4a3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bdb4299702d34d95978fa1a1059bc3ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c66468b224c84e6185599bcb57fcc0b1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a6ab4778fca3462eae88193b8ac79083, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e695c3c144fa4c50b03a9c24bb0fb637, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/19ef7ad9f3d64b47aee1fe3aa6ea1aae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/485ae869a6a94be7ad98e33862f0ceff, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb96a8d3ab1247b1855a1244da47456e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f90110bc25a24239bcc8edbf1f784e90, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0f778d2dcef84acc8a6c9c026f0efdbe, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de73afa187e8474a852e88461903895d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/8277233faf3b4fb7988e543a74005ae1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/69f89b453eb84540a722d3f7842be09d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/36efba515a694338a75a790b0ca33e14, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/cc3496a0c74d497aa1bcf80bbe0b6218, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c4dad6292d340d7ace470c24f4bb140, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de7176b532804a67b4aa2decef78281c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ea6272bcda6b4d8f9db9b181b2eac307, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca51fe62c52b4bd19313415f7e411482, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/02468db71b0548ae8b472aa0e415beaa, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a01c5b4a1d294d04a9b5073b9fcb53d8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/eec824fae6b24969bc15d277c1917d32, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/961a5e9ab5c6482f8470d55dace78dee, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ecc0ea0c84894b61a1f2a0cf829013e3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0b022825f0a3473db898914312eea06b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee741a76246c4f5cabf2eb3d709ec9b4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d757304f8c4c4bf8843b9373d32d771a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/94c84b4ca6b04b6aa11c9c879b255117, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee5aeb7a9ccc4aa284dddf4cac648a96] to archive 2023-03-22 17:56:15,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:56:15,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c4a1d79cce1d465792063315973694f5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c4a1d79cce1d465792063315973694f5 2023-03-22 17:56:15,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5ca93ffdaa914f83a60afdb71f66ad9c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5ca93ffdaa914f83a60afdb71f66ad9c 2023-03-22 17:56:15,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/6d034e6aabad47958089f784b4fb7b6c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/6d034e6aabad47958089f784b4fb7b6c 2023-03-22 17:56:15,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb9b0c4f73e445819a51a8618c4325ab to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb9b0c4f73e445819a51a8618c4325ab 2023-03-22 17:56:15,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d5e6586101754eaabec80e2e8b2b209a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d5e6586101754eaabec80e2e8b2b209a 2023-03-22 17:56:15,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca6c77931d5b45d2920917c762ea5779 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca6c77931d5b45d2920917c762ea5779 2023-03-22 17:56:15,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/320b138d0af3499288c641226014f2ec to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/320b138d0af3499288c641226014f2ec 2023-03-22 17:56:15,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/540b46a9293e4bf19ef77170e724462c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/540b46a9293e4bf19ef77170e724462c 2023-03-22 17:56:15,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/09c320cd9f8e46d08d93aa5999f62657 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/09c320cd9f8e46d08d93aa5999f62657 2023-03-22 17:56:15,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/fdde32930d0a443ca358fd7e04548df8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/fdde32930d0a443ca358fd7e04548df8 2023-03-22 17:56:15,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e443418e7b01422c848983ab896e10fa to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e443418e7b01422c848983ab896e10fa 2023-03-22 17:56:15,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a603ee386afa4c6cba9878c2d01acc2c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a603ee386afa4c6cba9878c2d01acc2c 2023-03-22 17:56:15,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d7c78d749c6843b6874c6e0ec477ee6d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d7c78d749c6843b6874c6e0ec477ee6d 2023-03-22 17:56:15,815 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5e52e6f3806747698cddb5183d35ec55 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/5e52e6f3806747698cddb5183d35ec55 2023-03-22 17:56:15,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d8e3d770c26c474882e549d8937bd57d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d8e3d770c26c474882e549d8937bd57d 2023-03-22 17:56:15,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e9a5ebcadd7b4c92b1bd8a77ab15000a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e9a5ebcadd7b4c92b1bd8a77ab15000a 2023-03-22 17:56:15,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f477e5d2c82c4ee185ab054fe73cd00c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f477e5d2c82c4ee185ab054fe73cd00c 2023-03-22 17:56:15,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/1eb4b5c4d1ea42d88fb4afcdbabc04f4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/1eb4b5c4d1ea42d88fb4afcdbabc04f4 2023-03-22 17:56:15,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c71a52dbe6b84a2f8055ae212130b646 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c71a52dbe6b84a2f8055ae212130b646 2023-03-22 17:56:15,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/651aa0e5fb50496c8b30295eeefaaefb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/651aa0e5fb50496c8b30295eeefaaefb 2023-03-22 17:56:15,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/9a5615ea28834320a63b10b1d27ca4a3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/9a5615ea28834320a63b10b1d27ca4a3 2023-03-22 17:56:15,821 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bdb4299702d34d95978fa1a1059bc3ae to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bdb4299702d34d95978fa1a1059bc3ae 2023-03-22 17:56:15,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c66468b224c84e6185599bcb57fcc0b1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/c66468b224c84e6185599bcb57fcc0b1 2023-03-22 17:56:15,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a6ab4778fca3462eae88193b8ac79083 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a6ab4778fca3462eae88193b8ac79083 2023-03-22 17:56:15,823 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e695c3c144fa4c50b03a9c24bb0fb637 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/e695c3c144fa4c50b03a9c24bb0fb637 2023-03-22 17:56:15,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/19ef7ad9f3d64b47aee1fe3aa6ea1aae to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/19ef7ad9f3d64b47aee1fe3aa6ea1aae 2023-03-22 17:56:15,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/485ae869a6a94be7ad98e33862f0ceff to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/485ae869a6a94be7ad98e33862f0ceff 2023-03-22 17:56:15,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb96a8d3ab1247b1855a1244da47456e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/bb96a8d3ab1247b1855a1244da47456e 2023-03-22 17:56:15,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f90110bc25a24239bcc8edbf1f784e90 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/f90110bc25a24239bcc8edbf1f784e90 2023-03-22 17:56:15,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0f778d2dcef84acc8a6c9c026f0efdbe to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0f778d2dcef84acc8a6c9c026f0efdbe 2023-03-22 17:56:15,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de73afa187e8474a852e88461903895d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de73afa187e8474a852e88461903895d 2023-03-22 17:56:15,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/8277233faf3b4fb7988e543a74005ae1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/8277233faf3b4fb7988e543a74005ae1 2023-03-22 17:56:15,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/69f89b453eb84540a722d3f7842be09d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/69f89b453eb84540a722d3f7842be09d 2023-03-22 17:56:15,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/36efba515a694338a75a790b0ca33e14 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/36efba515a694338a75a790b0ca33e14 2023-03-22 17:56:15,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/cc3496a0c74d497aa1bcf80bbe0b6218 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/cc3496a0c74d497aa1bcf80bbe0b6218 2023-03-22 17:56:15,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c4dad6292d340d7ace470c24f4bb140 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c4dad6292d340d7ace470c24f4bb140 2023-03-22 17:56:15,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de7176b532804a67b4aa2decef78281c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/de7176b532804a67b4aa2decef78281c 2023-03-22 17:56:15,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ea6272bcda6b4d8f9db9b181b2eac307 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ea6272bcda6b4d8f9db9b181b2eac307 2023-03-22 17:56:15,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca51fe62c52b4bd19313415f7e411482 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ca51fe62c52b4bd19313415f7e411482 2023-03-22 17:56:15,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/02468db71b0548ae8b472aa0e415beaa to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/02468db71b0548ae8b472aa0e415beaa 2023-03-22 17:56:15,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a01c5b4a1d294d04a9b5073b9fcb53d8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/a01c5b4a1d294d04a9b5073b9fcb53d8 2023-03-22 17:56:15,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/eec824fae6b24969bc15d277c1917d32 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/eec824fae6b24969bc15d277c1917d32 2023-03-22 17:56:15,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/961a5e9ab5c6482f8470d55dace78dee to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/961a5e9ab5c6482f8470d55dace78dee 2023-03-22 17:56:15,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ecc0ea0c84894b61a1f2a0cf829013e3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ecc0ea0c84894b61a1f2a0cf829013e3 2023-03-22 17:56:15,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0b022825f0a3473db898914312eea06b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0b022825f0a3473db898914312eea06b 2023-03-22 17:56:15,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee741a76246c4f5cabf2eb3d709ec9b4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee741a76246c4f5cabf2eb3d709ec9b4 2023-03-22 17:56:15,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d757304f8c4c4bf8843b9373d32d771a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/d757304f8c4c4bf8843b9373d32d771a 2023-03-22 17:56:15,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/94c84b4ca6b04b6aa11c9c879b255117 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/94c84b4ca6b04b6aa11c9c879b255117 2023-03-22 17:56:15,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee5aeb7a9ccc4aa284dddf4cac648a96 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ee5aeb7a9ccc4aa284dddf4cac648a96 2023-03-22 17:56:15,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/recovered.edits/843.seqid, newMaxSeqId=843, maxSeqId=1 2023-03-22 17:56:15,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282. 2023-03-22 17:56:15,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 523fd4cd7256d4e27c9367c2949e4282: 2023-03-22 17:56:15,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:15,848 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=523fd4cd7256d4e27c9367c2949e4282, regionState=CLOSED 2023-03-22 17:56:15,848 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507775848"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507775848"}]},"ts":"1679507775848"} 2023-03-22 17:56:15,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-22 17:56:15,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 523fd4cd7256d4e27c9367c2949e4282, server=jenkins-hbase4.apache.org,34403,1679507617723 in 1.6950 sec 2023-03-22 17:56:15,851 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-22 17:56:15,851 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=523fd4cd7256d4e27c9367c2949e4282, UNASSIGN in 1.6990 sec 2023-03-22 17:56:15,851 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507775851"}]},"ts":"1679507775851"} 2023-03-22 17:56:15,852 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-22 17:56:15,853 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-22 17:56:15,854 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8110 sec 2023-03-22 17:56:16,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-22 17:56:16,151 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-22 17:56:16,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-22 17:56:16,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:16,153 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:16,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-22 17:56:16,154 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:16,158 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:16,160 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/recovered.edits] 2023-03-22 17:56:16,162 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1eb1ebb31e2443d19b9c85959648fbed to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/1eb1ebb31e2443d19b9c85959648fbed 2023-03-22 17:56:16,163 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/82dcba2cfcfd4d05b8c91f7b7633ed7b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/82dcba2cfcfd4d05b8c91f7b7633ed7b 2023-03-22 17:56:16,164 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9185ebe2495b46e6a2cf421097ea5d89 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/9185ebe2495b46e6a2cf421097ea5d89 2023-03-22 17:56:16,165 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c816436145304850912777e0ca5ea514 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/A/c816436145304850912777e0ca5ea514 2023-03-22 17:56:16,167 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/40526d8a8c7544129f41d6782d93d518 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/40526d8a8c7544129f41d6782d93d518 2023-03-22 17:56:16,168 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/45ee5a0c558b48eea10e0af2e58004d0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/45ee5a0c558b48eea10e0af2e58004d0 2023-03-22 17:56:16,169 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/aacfde35eb2740bf897fc92e8c53c11a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/aacfde35eb2740bf897fc92e8c53c11a 2023-03-22 17:56:16,170 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f1487968a4d44c89bdea95e55ef1804a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/B/f1487968a4d44c89bdea95e55ef1804a 2023-03-22 17:56:16,172 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c1a680480d74a91a9920e951e20faea to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/0c1a680480d74a91a9920e951e20faea 2023-03-22 17:56:16,173 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/4858af5a4a0341fcb62d4411ab8b96bf to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/4858af5a4a0341fcb62d4411ab8b96bf 2023-03-22 17:56:16,173 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ccf80fc99ccc4269a7e329a161081145 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/C/ccf80fc99ccc4269a7e329a161081145 2023-03-22 17:56:16,175 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/recovered.edits/843.seqid to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282/recovered.edits/843.seqid 2023-03-22 17:56:16,176 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/523fd4cd7256d4e27c9367c2949e4282 2023-03-22 17:56:16,176 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:56:16,177 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:16,178 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-22 17:56:16,180 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-22 17:56:16,181 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:16,181 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-22 17:56:16,181 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679507776181"}]},"ts":"9223372036854775807"} 2023-03-22 17:56:16,182 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-22 17:56:16,182 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 523fd4cd7256d4e27c9367c2949e4282, NAME => 'TestAcidGuarantees,,1679507751406.523fd4cd7256d4e27c9367c2949e4282.', STARTKEY => '', ENDKEY => ''}] 2023-03-22 17:56:16,182 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-22 17:56:16,182 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679507776182"}]},"ts":"9223372036854775807"} 2023-03-22 17:56:16,183 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-22 17:56:16,185 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:16,186 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-03-22 17:56:16,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-22 17:56:16,255 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-22 17:56:16,267 INFO [Listener at localhost/44491] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=238 (was 226) - Thread LEAK? -, OpenFileDescriptor=637 (was 528) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=350 (was 308) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6175 (was 5852) - AvailableMemoryMB LEAK? - 2023-03-22 17:56:16,280 INFO [Listener at localhost/44491] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=238, OpenFileDescriptor=637, MaxFileDescriptor=60000, SystemLoadAverage=350, ProcessCount=169, AvailableMemoryMB=6174 2023-03-22 17:56:16,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:56:16,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:56:16,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:16,284 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-22 17:56:16,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-22 17:56:16,284 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-22 17:56:16,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-22 17:56:16,285 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 2023-03-22 17:56:16,286 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 empty. 2023-03-22 17:56:16,286 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 2023-03-22 17:56:16,286 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:56:16,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-22 17:56:16,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-22 17:56:16,694 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:56:16,695 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 876c388f3b400624afff0881af885f88, NAME => 'TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp 2023-03-22 17:56:16,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-22 17:56:17,100 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:56:17,101 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 876c388f3b400624afff0881af885f88, disabling compactions & flushes 2023-03-22 17:56:17,101 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:17,101 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:17,101 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. after waiting 0 ms 2023-03-22 17:56:17,101 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:17,101 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:17,101 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:17,103 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-22 17:56:17,103 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507777103"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507777103"}]},"ts":"1679507777103"} 2023-03-22 17:56:17,104 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-22 17:56:17,105 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-22 17:56:17,105 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507777105"}]},"ts":"1679507777105"} 2023-03-22 17:56:17,106 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-22 17:56:17,113 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, ASSIGN}] 2023-03-22 17:56:17,114 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, ASSIGN 2023-03-22 17:56:17,115 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=false 2023-03-22 17:56:17,266 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=876c388f3b400624afff0881af885f88, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:17,266 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507777266"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507777266"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507777266"}]},"ts":"1679507777266"} 2023-03-22 17:56:17,267 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:56:17,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-22 17:56:17,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:17,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 876c388f3b400624afff0881af885f88, NAME => 'TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:56:17,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:56:17,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,422 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,423 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A 2023-03-22 17:56:17,423 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A 2023-03-22 17:56:17,423 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:56:17,423 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 876c388f3b400624afff0881af885f88 columnFamilyName A 2023-03-22 17:56:17,424 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(310): Store=876c388f3b400624afff0881af885f88/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:56:17,424 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,425 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B 2023-03-22 17:56:17,425 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B 2023-03-22 17:56:17,425 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:56:17,425 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 876c388f3b400624afff0881af885f88 columnFamilyName B 2023-03-22 17:56:17,426 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(310): Store=876c388f3b400624afff0881af885f88/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:56:17,426 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,427 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C 2023-03-22 17:56:17,427 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C 2023-03-22 17:56:17,427 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:56:17,427 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 876c388f3b400624afff0881af885f88 columnFamilyName C 2023-03-22 17:56:17,427 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(310): Store=876c388f3b400624afff0881af885f88/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:56:17,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:17,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:56:17,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 876c388f3b400624afff0881af885f88 2023-03-22 17:56:17,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-22 17:56:17,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 876c388f3b400624afff0881af885f88; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66241356, jitterRate=-0.012926876544952393}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:56:17,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:17,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., pid=56, masterSystemTime=1679507777418 2023-03-22 17:56:17,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:17,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:17,434 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=876c388f3b400624afff0881af885f88, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:17,434 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507777434"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507777434"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507777434"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507777434"}]},"ts":"1679507777434"} 2023-03-22 17:56:17,436 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-22 17:56:17,436 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 in 168 msec 2023-03-22 17:56:17,437 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-22 17:56:17,437 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, ASSIGN in 323 msec 2023-03-22 17:56:17,437 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-22 17:56:17,438 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507777437"}]},"ts":"1679507777437"} 2023-03-22 17:56:17,438 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-22 17:56:17,440 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-22 17:56:17,441 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-03-22 17:56:18,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-22 17:56:18,389 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-22 17:56:18,390 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:18,395 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:18,396 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:18,397 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43454, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:18,398 DEBUG [Listener at localhost/44491] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-22 17:56:18,399 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57432, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-22 17:56:18,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-22 17:56:18,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-22 17:56:18,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:18,816 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-22 17:56:18,816 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-22 17:56:18,817 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-22 17:56:18,818 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-22 17:56:18,820 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, REOPEN/MOVE}] 2023-03-22 17:56:18,821 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, REOPEN/MOVE 2023-03-22 17:56:18,822 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=876c388f3b400624afff0881af885f88, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:18,822 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507778821"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507778821"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507778821"}]},"ts":"1679507778821"} 2023-03-22 17:56:18,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:56:18,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 876c388f3b400624afff0881af885f88 2023-03-22 17:56:18,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 876c388f3b400624afff0881af885f88, disabling compactions & flushes 2023-03-22 17:56:18,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:18,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:18,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. after waiting 0 ms 2023-03-22 17:56:18,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:18,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-22 17:56:18,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:18,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:18,982 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 876c388f3b400624afff0881af885f88 to self. 2023-03-22 17:56:18,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 876c388f3b400624afff0881af885f88 2023-03-22 17:56:18,983 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=876c388f3b400624afff0881af885f88, regionState=CLOSED 2023-03-22 17:56:18,983 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507778983"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507778983"}]},"ts":"1679507778983"} 2023-03-22 17:56:18,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-22 17:56:18,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 in 161 msec 2023-03-22 17:56:18,985 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34403,1679507617723; forceNewPlan=false, retain=true 2023-03-22 17:56:19,136 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=876c388f3b400624afff0881af885f88, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,136 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507779136"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507779136"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507779136"}]},"ts":"1679507779136"} 2023-03-22 17:56:19,137 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:56:19,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:19,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 876c388f3b400624afff0881af885f88, NAME => 'TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.', STARTKEY => '', ENDKEY => ''} 2023-03-22 17:56:19,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-22 17:56:19,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,294 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,294 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A 2023-03-22 17:56:19,294 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A 2023-03-22 17:56:19,295 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:56:19,295 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 876c388f3b400624afff0881af885f88 columnFamilyName A 2023-03-22 17:56:19,297 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(310): Store=876c388f3b400624afff0881af885f88/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:56:19,297 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,297 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B 2023-03-22 17:56:19,298 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B 2023-03-22 17:56:19,298 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:56:19,298 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 876c388f3b400624afff0881af885f88 columnFamilyName B 2023-03-22 17:56:19,298 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(310): Store=876c388f3b400624afff0881af885f88/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:56:19,298 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,299 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C 2023-03-22 17:56:19,299 DEBUG [StoreOpener-876c388f3b400624afff0881af885f88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C 2023-03-22 17:56:19,299 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-22 17:56:19,299 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 876c388f3b400624afff0881af885f88 columnFamilyName C 2023-03-22 17:56:19,299 INFO [StoreOpener-876c388f3b400624afff0881af885f88-1] regionserver.HStore(310): Store=876c388f3b400624afff0881af885f88/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-22 17:56:19,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:19,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-22 17:56:19,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 876c388f3b400624afff0881af885f88; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62196567, jitterRate=-0.07319892942905426}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-22 17:56:19,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:19,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., pid=61, masterSystemTime=1679507779289 2023-03-22 17:56:19,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:19,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:19,306 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=876c388f3b400624afff0881af885f88, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,307 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507779306"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679507779306"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679507779306"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679507779306"}]},"ts":"1679507779306"} 2023-03-22 17:56:19,308 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-22 17:56:19,308 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 in 170 msec 2023-03-22 17:56:19,309 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-22 17:56:19,309 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, REOPEN/MOVE in 488 msec 2023-03-22 17:56:19,311 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-22 17:56:19,311 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-03-22 17:56:19,312 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 910 msec 2023-03-22 17:56:19,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-22 17:56:19,314 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,318 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,319 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,322 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,322 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,327 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,328 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,331 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,332 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,335 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,336 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,342 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,342 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,347 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,348 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,353 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,354 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,361 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,361 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(139): Connect 0x0eedd898 to 127.0.0.1:55285 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-22 17:56:19,365 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@308b3f39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-22 17:56:19,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:19,376 DEBUG [hconnection-0x66a30168-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,383 DEBUG [hconnection-0x35634efe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,383 DEBUG [hconnection-0x7a0b4ab4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,383 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43456, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,383 DEBUG [hconnection-0x2ffc3a61-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,384 DEBUG [hconnection-0x6613b3e1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,384 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43460, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,384 DEBUG [hconnection-0x383df43e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,385 DEBUG [hconnection-0x3093e03c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,386 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43470, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,385 DEBUG [hconnection-0x125b6eac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,386 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43504, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:19,389 DEBUG [hconnection-0x31872ac8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:19,391 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:19,391 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:19,389 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43492, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,388 DEBUG [hconnection-0x78c073e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-22 17:56:19,391 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:19,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:19,391 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43486, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,393 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43538, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,393 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,397 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43520, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:19,399 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43550, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-22 17:56:19,400 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:19,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:19,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,402 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,402 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:19,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:19,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:19,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:19,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:19,406 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:19,406 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:19,407 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-22 17:56:19,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:19,408 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:19,408 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:19,408 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:19,408 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:19,408 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:19,408 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:19,409 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:19,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507839420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507839421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507779415, max timestamp=1679507779422Num uniques 859; 2023-03-22 17:56:19,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:19,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507779415, max timestamp=1679507779422Num uniques 859; 2023-03-22 17:56:19,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:19,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507779415, max timestamp=1679507779422Num uniques 859; 2023-03-22 17:56:19,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:19,433 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322b06442b52e3941d99fa6a9dfa3826e31 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b06442b52e3941d99fa6a9dfa3826e31 2023-03-22 17:56:19,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507839441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507839442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507839443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507839524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507839525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507839544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507839547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507839547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507839727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507839728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507839746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507839750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:19,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507839751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,839 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/305f919884a74173b71561debf486d93 2023-03-22 17:56:19,848 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/0115929ff37d4bbf938134ccff7256a9 2023-03-22 17:56:19,856 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/2db51ca0ea274d4383180b2323631769 2023-03-22 17:56:19,860 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/305f919884a74173b71561debf486d93 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/305f919884a74173b71561debf486d93 2023-03-22 17:56:19,863 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/305f919884a74173b71561debf486d93, entries=150, sequenceid=15, filesize=25.1 K 2023-03-22 17:56:19,863 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/0115929ff37d4bbf938134ccff7256a9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0115929ff37d4bbf938134ccff7256a9 2023-03-22 17:56:19,866 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0115929ff37d4bbf938134ccff7256a9, entries=100, sequenceid=15, filesize=9.3 K 2023-03-22 17:56:19,867 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/2db51ca0ea274d4383180b2323631769 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2db51ca0ea274d4383180b2323631769 2023-03-22 17:56:19,870 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2db51ca0ea274d4383180b2323631769, entries=100, sequenceid=15, filesize=9.3 K 2023-03-22 17:56:19,871 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 876c388f3b400624afff0881af885f88 in 464ms, sequenceid=15, compaction requested=false 2023-03-22 17:56:19,871 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-22 17:56:19,872 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:19,872 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:19,876 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:19,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:19,876 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:19,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:19,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f77181c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:19,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:19,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:19,880 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:19,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:19,880 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,880 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:19,882 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,882 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:19,882 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:19,882 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:19,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:19,885 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:19,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:19,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:19,885 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:19,885 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:19,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:19,888 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,888 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:19,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:19,888 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:19,890 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:19,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:19,893 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:19,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:19,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:19,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:19,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:19,894 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:19,894 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:19,994 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:19,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:19,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:19,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:19,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:19,997 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:19,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:19,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:19,997 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:19,998 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:19,998 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:19,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,999 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:19,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:19,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:19,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:20,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:20,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:20,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:20,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:20,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:20,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:20,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:20,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:20,001 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:20,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:20,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:20,001 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-03-22 17:56:20,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:20,001 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:20,002 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:20,002 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:20,002 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:20,002 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:20,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:20,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507840046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507840047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507840052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507840053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507840054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507840150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507840150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507840353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507840353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,421 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322129bc996ae9941878d3ed267f3996225 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322129bc996ae9941878d3ed267f3996225 2023-03-22 17:56:20,426 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/1052d347c76c4f17b9dde0f7d8e0521d 2023-03-22 17:56:20,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507840555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507840557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507840562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507840658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:20,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507840658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:20,837 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/f5a988776c0e4b3297dd8533d3a55606 2023-03-22 17:56:21,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507841162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507841162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,251 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/db590b3d18d54dcba3b319790f42875b 2023-03-22 17:56:21,256 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/1052d347c76c4f17b9dde0f7d8e0521d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1052d347c76c4f17b9dde0f7d8e0521d 2023-03-22 17:56:21,259 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1052d347c76c4f17b9dde0f7d8e0521d, entries=150, sequenceid=51, filesize=25.1 K 2023-03-22 17:56:21,260 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/f5a988776c0e4b3297dd8533d3a55606 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f5a988776c0e4b3297dd8533d3a55606 2023-03-22 17:56:21,263 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f5a988776c0e4b3297dd8533d3a55606, entries=150, sequenceid=51, filesize=11.6 K 2023-03-22 17:56:21,264 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/db590b3d18d54dcba3b319790f42875b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/db590b3d18d54dcba3b319790f42875b 2023-03-22 17:56:21,267 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/db590b3d18d54dcba3b319790f42875b, entries=150, sequenceid=51, filesize=11.6 K 2023-03-22 17:56:21,268 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for 876c388f3b400624afff0881af885f88 in 1266ms, sequenceid=51, compaction requested=false 2023-03-22 17:56:21,268 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:21,268 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:21,278 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,278 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:21,278 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:21,278 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8fe7791[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:21,280 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:21,280 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:21,281 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:21,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:21,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:21,282 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:21,282 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:21,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,282 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:21,285 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:21,285 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:21,285 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:21,285 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:21,285 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:21,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:21,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,287 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:21,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,287 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:21,289 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:21,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:21,292 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,292 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:21,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:21,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:21,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:21,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:21,293 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:21,293 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:21,379 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-22 17:56:21,393 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:21,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:21,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:21,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:21,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:21,402 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:21,402 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:21,402 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:21,402 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:21,402 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,402 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:21,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,404 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:21,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:21,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:21,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:21,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:21,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:21,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:21,406 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:21,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:21,420 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:56:21,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:21,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:21,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:21,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:21,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:21,420 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:21,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:21,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:21,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507781599, max timestamp=1679507781609Num uniques 859; 2023-03-22 17:56:21,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:21,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507781599, max timestamp=1679507781609Num uniques 859; 2023-03-22 17:56:21,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:21,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507781599, max timestamp=1679507781609Num uniques 859; 2023-03-22 17:56:21,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:21,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507841634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507841637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507841638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507841742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507841743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507841743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,869 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322b8c1ab014cf242a2b8342161bfaeeb12 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b8c1ab014cf242a2b8342161bfaeeb12 2023-03-22 17:56:21,873 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/80453f10dd164ec08ae49c3d874c499a 2023-03-22 17:56:21,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507841945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507841948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:21,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:21,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507841949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507842168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507842170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507842248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507842253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507842254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,282 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/9a85eab2cc704fe7bc47baa83f27d7ae 2023-03-22 17:56:22,692 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/785cd1f4021749559c455e07ccdbab16 2023-03-22 17:56:22,698 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/80453f10dd164ec08ae49c3d874c499a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/80453f10dd164ec08ae49c3d874c499a 2023-03-22 17:56:22,701 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/80453f10dd164ec08ae49c3d874c499a, entries=150, sequenceid=62, filesize=25.1 K 2023-03-22 17:56:22,702 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/9a85eab2cc704fe7bc47baa83f27d7ae as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9a85eab2cc704fe7bc47baa83f27d7ae 2023-03-22 17:56:22,705 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9a85eab2cc704fe7bc47baa83f27d7ae, entries=150, sequenceid=62, filesize=11.6 K 2023-03-22 17:56:22,706 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/785cd1f4021749559c455e07ccdbab16 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/785cd1f4021749559c455e07ccdbab16 2023-03-22 17:56:22,709 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/785cd1f4021749559c455e07ccdbab16, entries=150, sequenceid=62, filesize=11.6 K 2023-03-22 17:56:22,710 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 876c388f3b400624afff0881af885f88 in 1290ms, sequenceid=62, compaction requested=true 2023-03-22 17:56:22,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:22,710 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:22,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,712 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,712 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76ad9cdc[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:22,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:22,714 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,715 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:22,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,715 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:22,717 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:22,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:22,720 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:22,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:22,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:22,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:22,720 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:22,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:22,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,721 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,722 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,723 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:22,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:22,726 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:22,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:22,726 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:22,726 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,726 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:22,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:22,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:22,756 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-22 17:56:22,756 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:22,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:22,756 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:22,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:22,756 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:22,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:22,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507842765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507842766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507842766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,771 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322e9b5459d30ab47d5a79e949105a8ac17 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e9b5459d30ab47d5a79e949105a8ac17 2023-03-22 17:56:22,775 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/00f47cafd5274e95b3056b786fb6dd2b 2023-03-22 17:56:22,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/80f513a154e249ea983f9b72444cfbf7 2023-03-22 17:56:22,827 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:22,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:22,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:22,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:22,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:22,830 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:22,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:22,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:22,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:22,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,830 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,831 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,831 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:22,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:22,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,832 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:22,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:22,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:22,833 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:22,833 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:22,833 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:22,833 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:22,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:22,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:22,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:22,834 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:22,836 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:22,838 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:22,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14faea66[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:22,838 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,839 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:22,839 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,839 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:22,841 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:22,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:22,843 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:22,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:22,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:22,843 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:22,843 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:22,845 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,845 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,845 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,847 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:22,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:22,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:22,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:22,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:22,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:22,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:22,850 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:22,851 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:22,851 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:22,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507842870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507842870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:22,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507842870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,951 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:22,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:22,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:22,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:22,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:22,955 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:22,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:22,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:22,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:22,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,956 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:22,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:22,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:22,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:22,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:22,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:22,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:22,958 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:22,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530ab71d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:22,962 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:22,962 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,973 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:22,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:22,973 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:22,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,973 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:22,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:22,974 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:22,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,975 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:22,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:22,977 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:22,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:22,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:22,977 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:22,977 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:22,980 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,980 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,980 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:22,981 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:22,982 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:22,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:22,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:22,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:22,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:22,984 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:22,985 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:22,984 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:22,985 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:22,985 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:22,985 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:22,985 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:22,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:23,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507843073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507843073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507843073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,085 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:23,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:23,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:23,089 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:23,089 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:23,089 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:23,089 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:23,089 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,089 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,091 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:23,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:23,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:23,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:23,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:23,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:23,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:23,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:23,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:23,093 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,094 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:23,094 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:23,094 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:23,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:23,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:23,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:23,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,095 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:23,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28331012[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:23,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,099 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:23,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:23,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:23,099 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,099 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:23,100 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:23,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:23,100 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:23,103 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:23,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:23,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:23,103 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:23,103 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:23,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,105 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,107 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,109 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,110 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,110 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:23,110 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:23,110 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:23,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:23,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:23,110 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:23,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:23,110 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:23,110 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:23,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:23,205 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/68f4604c1daf4423bb11e6660444e705 2023-03-22 17:56:23,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/00f47cafd5274e95b3056b786fb6dd2b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/00f47cafd5274e95b3056b786fb6dd2b 2023-03-22 17:56:23,211 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:23,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:23,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:23,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/00f47cafd5274e95b3056b786fb6dd2b, entries=200, sequenceid=99, filesize=31.9 K 2023-03-22 17:56:23,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:23,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:23,215 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:23,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:23,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:23,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:23,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/80f513a154e249ea983f9b72444cfbf7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/80f513a154e249ea983f9b72444cfbf7 2023-03-22 17:56:23,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,215 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,217 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,217 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:23,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:23,217 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,217 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:23,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:23,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:23,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:23,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:23,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/80f513a154e249ea983f9b72444cfbf7, entries=150, sequenceid=99, filesize=11.6 K 2023-03-22 17:56:23,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:23,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:23,218 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:23,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:23,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:23,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:23,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/68f4604c1daf4423bb11e6660444e705 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/68f4604c1daf4423bb11e6660444e705 2023-03-22 17:56:23,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:23,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:23,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:23,221 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/68f4604c1daf4423bb11e6660444e705, entries=150, sequenceid=99, filesize=11.6 K 2023-03-22 17:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:23,223 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 876c388f3b400624afff0881af885f88 in 467ms, sequenceid=99, compaction requested=true 2023-03-22 17:56:23,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:23,223 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:23,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@326232b4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:23,223 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:23,223 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:23,223 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:23,223 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:23,223 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:23,224 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109920 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:23,224 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:23,224 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:23,225 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:23,225 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,225 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,225 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/305f919884a74173b71561debf486d93, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1052d347c76c4f17b9dde0f7d8e0521d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/80453f10dd164ec08ae49c3d874c499a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/00f47cafd5274e95b3056b786fb6dd2b] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=107.3 K 2023-03-22 17:56:23,225 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0115929ff37d4bbf938134ccff7256a9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f5a988776c0e4b3297dd8533d3a55606, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9a85eab2cc704fe7bc47baa83f27d7ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/80f513a154e249ea983f9b72444cfbf7] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=44.1 K 2023-03-22 17:56:23,225 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:23,225 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0115929ff37d4bbf938134ccff7256a9, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679507779406 2023-03-22 17:56:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:23,225 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 305f919884a74173b71561debf486d93, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679507779406 2023-03-22 17:56:23,226 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:23,226 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f5a988776c0e4b3297dd8533d3a55606, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507779437 2023-03-22 17:56:23,226 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1052d347c76c4f17b9dde0f7d8e0521d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507779437 2023-03-22 17:56:23,226 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9a85eab2cc704fe7bc47baa83f27d7ae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679507780032 2023-03-22 17:56:23,226 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 80453f10dd164ec08ae49c3d874c499a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679507780032 2023-03-22 17:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:23,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,227 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 80f513a154e249ea983f9b72444cfbf7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679507781633 2023-03-22 17:56:23,227 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 00f47cafd5274e95b3056b786fb6dd2b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679507781633 2023-03-22 17:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:23,231 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:23,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:23,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:23,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:23,231 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:23,233 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,233 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:23,233 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,234 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,235 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,237 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#544 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,237 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,238 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:23,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:23,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:23,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:23,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:23,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:23,238 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:23,238 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:23,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:23,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:23,243 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#543 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:23,266 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/5492ba0b905e49dcaf46cb95b805a88c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/5492ba0b905e49dcaf46cb95b805a88c 2023-03-22 17:56:23,270 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into 5492ba0b905e49dcaf46cb95b805a88c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:23,270 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:23,270 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=12, startTime=1679507783223; duration=0sec 2023-03-22 17:56:23,271 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:23,271 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:23,272 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:23,272 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:23,272 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,272 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2db51ca0ea274d4383180b2323631769, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/db590b3d18d54dcba3b319790f42875b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/785cd1f4021749559c455e07ccdbab16, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/68f4604c1daf4423bb11e6660444e705] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=44.1 K 2023-03-22 17:56:23,273 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2db51ca0ea274d4383180b2323631769, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679507779406 2023-03-22 17:56:23,273 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting db590b3d18d54dcba3b319790f42875b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679507779437 2023-03-22 17:56:23,274 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 785cd1f4021749559c455e07ccdbab16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679507780032 2023-03-22 17:56:23,274 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 68f4604c1daf4423bb11e6660444e705, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679507781633 2023-03-22 17:56:23,281 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#545 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:23,297 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/7c5931d587cc4cfda78ba6e128321e70 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7c5931d587cc4cfda78ba6e128321e70 2023-03-22 17:56:23,302 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into 7c5931d587cc4cfda78ba6e128321e70(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:23,302 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:23,302 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=12, startTime=1679507783223; duration=0sec 2023-03-22 17:56:23,302 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:23,338 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:23,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:23,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:23,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:23,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:23,341 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:23,341 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:23,341 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:23,341 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:23,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:23,344 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:23,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:23,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:23,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:23,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:23,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:23,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:23,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:23,346 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,346 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:23,346 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-22 17:56:23,346 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:23,346 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:23,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:23,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:23,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:23,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:23,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:23,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:23,382 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032218507229980141b4a9cd396cd96c52c3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032218507229980141b4a9cd396cd96c52c3 2023-03-22 17:56:23,387 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/f66ce7060a5643a985c0b1a9966330a1 2023-03-22 17:56:23,396 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/bba7a583f83b409faef865fcaab4b88a 2023-03-22 17:56:23,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507783391, max timestamp=1679507783403Num uniques 859; 2023-03-22 17:56:23,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507783391, max timestamp=1679507783403Num uniques 859; 2023-03-22 17:56:23,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679507783391, max timestamp=1679507783403Num uniques 859; 2023-03-22 17:56:23,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:56:23,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:56:23,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-22 17:56:23,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507843425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507843427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507843427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507843528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507843530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507843531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,682 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/f45a8cf5641e4074b846f5abbe4a1dfd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f45a8cf5641e4074b846f5abbe4a1dfd 2023-03-22 17:56:23,686 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into f45a8cf5641e4074b846f5abbe4a1dfd(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:23,686 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:23,686 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=12, startTime=1679507783223; duration=0sec 2023-03-22 17:56:23,687 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:23,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507843731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507843734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:23,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507843734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,815 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/c94275372931452d862997cabc0ca6f0 2023-03-22 17:56:23,818 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/f66ce7060a5643a985c0b1a9966330a1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f66ce7060a5643a985c0b1a9966330a1 2023-03-22 17:56:23,822 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f66ce7060a5643a985c0b1a9966330a1, entries=150, sequenceid=110, filesize=25.1 K 2023-03-22 17:56:23,823 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/bba7a583f83b409faef865fcaab4b88a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bba7a583f83b409faef865fcaab4b88a 2023-03-22 17:56:23,826 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bba7a583f83b409faef865fcaab4b88a, entries=150, sequenceid=110, filesize=11.6 K 2023-03-22 17:56:23,827 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/c94275372931452d862997cabc0ca6f0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c94275372931452d862997cabc0ca6f0 2023-03-22 17:56:23,830 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c94275372931452d862997cabc0ca6f0, entries=150, sequenceid=110, filesize=11.6 K 2023-03-22 17:56:23,830 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 876c388f3b400624afff0881af885f88 in 484ms, sequenceid=110, compaction requested=false 2023-03-22 17:56:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:23,831 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:23,831 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:23,832 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:23,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71885744[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:23,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:23,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,835 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:23,835 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:23,836 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,836 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:23,843 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,844 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:23,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:23,846 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:23,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:23,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:23,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:23,846 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:23,847 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:23,847 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,847 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:23,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:23,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:23,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,849 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:23,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,851 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:23,851 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:23,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:23,851 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:23,851 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:23,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:23,952 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:23,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:23,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:23,955 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:23,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:23,955 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:23,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:23,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,956 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,957 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:23,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:23,958 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:23,958 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:23,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:23,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:23,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:23,959 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:23,959 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:23,959 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:23,959 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-22 17:56:23,960 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:23,960 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:23,960 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:23,960 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:23,960 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:23,960 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:23,979 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032299f87b653d3b4c5d935008cc6baa3e8d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032299f87b653d3b4c5d935008cc6baa3e8d 2023-03-22 17:56:23,984 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/400005421946496dad969d5d38089601 2023-03-22 17:56:23,996 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/ec4b2cacc25748129477f4324bd33246 2023-03-22 17:56:24,020 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/977f19a222f34be69142b2543a86bdf3 2023-03-22 17:56:24,026 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/400005421946496dad969d5d38089601 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/400005421946496dad969d5d38089601 2023-03-22 17:56:24,030 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/400005421946496dad969d5d38089601, entries=150, sequenceid=147, filesize=25.3 K 2023-03-22 17:56:24,031 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/ec4b2cacc25748129477f4324bd33246 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/ec4b2cacc25748129477f4324bd33246 2023-03-22 17:56:24,034 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/ec4b2cacc25748129477f4324bd33246, entries=150, sequenceid=147, filesize=11.8 K 2023-03-22 17:56:24,034 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/977f19a222f34be69142b2543a86bdf3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/977f19a222f34be69142b2543a86bdf3 2023-03-22 17:56:24,046 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/977f19a222f34be69142b2543a86bdf3, entries=150, sequenceid=147, filesize=11.8 K 2023-03-22 17:56:24,046 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=0 B/0 for 876c388f3b400624afff0881af885f88 in 87ms, sequenceid=147, compaction requested=true 2023-03-22 17:56:24,046 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:24,046 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:24,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77bca437[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:24,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:24,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,053 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:24,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:24,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:24,053 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,053 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:24,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:24,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:24,058 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:24,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:24,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:24,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:24,058 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:24,060 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,060 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,060 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,060 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:24,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:24,065 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:24,065 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:24,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:24,083 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:24,084 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:24,084 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:24,084 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:24,084 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:24,084 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:24,084 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:24,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507784106, max timestamp=1679507784111Num uniques 859; 2023-03-22 17:56:24,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507784106, max timestamp=1679507784111Num uniques 859; 2023-03-22 17:56:24,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:24,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507784106, max timestamp=1679507784111Num uniques 859; 2023-03-22 17:56:24,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:24,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:24,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507844135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507844136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507844136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,165 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:24,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:24,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:24,170 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:24,170 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,170 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:24,170 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:24,170 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,170 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,172 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,172 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:24,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:24,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:24,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:24,179 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,179 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:24,179 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:24,179 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:24,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,181 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507844179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:24,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473f9952[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:24,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,184 DEBUG [Thread-4005] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:56:24,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:24,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:24,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:24,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,186 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:24,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507844187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,188 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,188 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:24,188 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:24,189 DEBUG [Thread-4013] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:56:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:24,193 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:24,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:24,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:24,193 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:24,193 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:24,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,196 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,198 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:24,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:24,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:24,201 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:24,201 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:24,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507844240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507844241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507844241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,301 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:24,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:24,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:24,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:24,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:24,305 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:24,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:24,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:24,306 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,306 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:24,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:24,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:24,310 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:24,310 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,310 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:24,310 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:24,310 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:24,310 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:24,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:24,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:24,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:24,313 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:24,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:24,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646bf493[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:24,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:24,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:24,318 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:24,319 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,319 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:24,320 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:24,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:24,324 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:24,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:24,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:24,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:24,324 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:24,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,328 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,329 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:24,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:24,341 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:24,341 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:24,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:24,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,441 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:24,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:24,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:24,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:24,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:24,444 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:24,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:24,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:24,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507844442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507844445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507844445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,449 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,449 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:24,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:24,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:24,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:24,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:24,450 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:24,450 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,450 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:24,451 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:24,451 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:24,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:24,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:24,451 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:24,453 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,453 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,453 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,453 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57eaad2d[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:24,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:24,454 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,455 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:24,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:24,456 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:24,456 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,456 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:24,458 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,458 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:24,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,459 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:24,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:24,461 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:24,461 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:24,461 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:24,461 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:24,461 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:24,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:24,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,467 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,468 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:24,468 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:24,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,503 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032208a6229d402148afafc6f61910d2c972 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032208a6229d402148afafc6f61910d2c972 2023-03-22 17:56:24,508 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/c7bb560d59304908b798310e80b3c46d 2023-03-22 17:56:24,568 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:24,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:24,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:24,571 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:24,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:24,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:24,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,571 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,573 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,573 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,573 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,573 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:24,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:24,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:24,576 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,576 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:24,576 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:24,576 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:24,579 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,579 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,580 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,580 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e526fb[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:24,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:24,582 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,583 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:24,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:24,583 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:24,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,583 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:24,584 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:24,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,584 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:24,587 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:24,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:24,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:24,587 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:24,587 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:24,589 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,589 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,589 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,590 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,592 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,596 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,597 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:24,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:24,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:24,597 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,597 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:24,598 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:24,698 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:24,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:24,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:24,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:24,701 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:24,701 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:24,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:24,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,702 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,703 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,703 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,703 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,703 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:24,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:24,705 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,705 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:24,705 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:24,705 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:24,706 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,706 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:24,706 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:24,707 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,707 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,708 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eabc21c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:24,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:24,710 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,711 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:24,711 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:24,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:24,711 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:24,714 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,714 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:24,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:24,716 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:24,716 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:24,716 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:24,716 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:24,716 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:24,718 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,718 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,718 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,719 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,720 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,723 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,723 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,723 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,723 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:24,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:24,723 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:24,723 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:24,723 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,723 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,723 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,724 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,724 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,724 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507844747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507844748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:24,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507844749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,823 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:24,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:24,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:24,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:24,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:24,827 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:24,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:24,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:24,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,834 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,834 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,835 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:24,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:24,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:24,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:24,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:24,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:24,836 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:24,836 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,836 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:24,836 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:24,837 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:24,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:24,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:24,837 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:24,838 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,839 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45df6e36[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:24,840 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:24,840 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,841 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,842 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:24,842 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:24,842 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:24,842 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,842 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:24,843 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,843 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,843 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,843 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:24,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:24,845 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:24,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:24,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:24,846 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:24,846 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:24,848 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,848 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,848 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,848 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,850 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,853 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,853 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:24,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:24,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:24,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,854 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:24,854 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:24,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/299bf2f4e3504270a8254e96963fa6f7 2023-03-22 17:56:24,954 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:24,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:24,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:24,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:24,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:24,957 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:24,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:24,957 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:24,958 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,958 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,960 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,960 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,960 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,960 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:24,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:24,961 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:24,961 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:24,961 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:24,962 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,962 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:24,962 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:24,962 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:24,964 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,964 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:24,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@475e3065[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:24,965 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:24,966 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,968 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:24,968 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,968 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:24,968 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:24,968 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,970 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:24,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:24,972 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:24,972 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:24,972 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:24,972 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:24,973 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:24,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:24,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,974 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,975 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:24,976 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:24,978 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:24,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:24,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:24,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:24,978 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:24,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:24,979 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:24,979 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:24,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:25,079 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:25,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:25,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:25,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:25,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:25,082 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:25,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:25,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:25,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:25,082 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,083 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,085 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,085 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:25,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:25,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:25,085 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,086 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:25,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:25,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:25,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:25,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:25,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:25,087 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:25,088 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,088 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:25,089 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,089 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:25,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:25,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b60048b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:25,090 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:25,090 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,092 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,092 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:25,092 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:25,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,092 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:25,092 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:25,093 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:25,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:25,097 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:25,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:25,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:25,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:25,097 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:25,100 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,100 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:25,100 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:25,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:25,100 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:25,102 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:25,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:25,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:25,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:25,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:25,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:25,104 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,105 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:25,105 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:25,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:25,205 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:25,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:25,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:25,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:25,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:25,209 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:25,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:25,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:25,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:25,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,209 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,213 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:25,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:25,214 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,214 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:25,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:25,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:25,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:25,215 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:25,216 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:25,216 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:25,216 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:25,219 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:25,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:25,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,219 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:25,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:25,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bfb98d9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:25,221 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:25,221 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,222 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:25,222 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,223 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:25,225 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:25,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:25,225 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:25,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:25,228 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:25,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:25,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:25,228 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:25,228 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:25,230 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:25,230 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,230 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:25,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,232 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,235 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:25,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:25,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:25,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:25,236 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:25,236 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:25,235 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,236 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:25,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:25,236 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,236 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:25,236 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,236 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,236 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,236 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:25,236 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:25,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507845252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:25,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507845253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:25,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507845256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/73afeff9f0db44429968c30412653ec2 2023-03-22 17:56:25,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/c7bb560d59304908b798310e80b3c46d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c7bb560d59304908b798310e80b3c46d 2023-03-22 17:56:25,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c7bb560d59304908b798310e80b3c46d, entries=400, sequenceid=158, filesize=59.4 K 2023-03-22 17:56:25,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/299bf2f4e3504270a8254e96963fa6f7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/299bf2f4e3504270a8254e96963fa6f7 2023-03-22 17:56:25,336 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:25,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:25,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/299bf2f4e3504270a8254e96963fa6f7, entries=150, sequenceid=158, filesize=11.8 K 2023-03-22 17:56:25,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:25,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/73afeff9f0db44429968c30412653ec2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/73afeff9f0db44429968c30412653ec2 2023-03-22 17:56:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:25,340 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:25,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:25,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:25,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:25,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,340 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,342 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:25,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:25,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:25,342 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:25,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/73afeff9f0db44429968c30412653ec2, entries=150, sequenceid=158, filesize=11.8 K 2023-03-22 17:56:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:25,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 876c388f3b400624afff0881af885f88 in 1260ms, sequenceid=158, compaction requested=true 2023-03-22 17:56:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:25,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:25,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:25,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:25,343 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:25,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:25,343 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:25,344 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138380 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:25,345 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:25,345 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:25,345 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f45a8cf5641e4074b846f5abbe4a1dfd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f66ce7060a5643a985c0b1a9966330a1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/400005421946496dad969d5d38089601, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c7bb560d59304908b798310e80b3c46d] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=135.1 K 2023-03-22 17:56:25,345 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:25,345 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:25,345 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:25,345 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/5492ba0b905e49dcaf46cb95b805a88c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bba7a583f83b409faef865fcaab4b88a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/ec4b2cacc25748129477f4324bd33246, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/299bf2f4e3504270a8254e96963fa6f7] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=46.9 K 2023-03-22 17:56:25,345 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f45a8cf5641e4074b846f5abbe4a1dfd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679507781633 2023-03-22 17:56:25,346 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 5492ba0b905e49dcaf46cb95b805a88c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679507781633 2023-03-22 17:56:25,346 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f66ce7060a5643a985c0b1a9966330a1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679507782760 2023-03-22 17:56:25,346 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting bba7a583f83b409faef865fcaab4b88a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679507782760 2023-03-22 17:56:25,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:25,346 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 400005421946496dad969d5d38089601, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679507783422 2023-03-22 17:56:25,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:25,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:25,346 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting ec4b2cacc25748129477f4324bd33246, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679507783422 2023-03-22 17:56:25,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:25,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:25,346 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:25,346 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c7bb560d59304908b798310e80b3c46d, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507784070 2023-03-22 17:56:25,347 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 299bf2f4e3504270a8254e96963fa6f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507784071 2023-03-22 17:56:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:25,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:25,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:25,347 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-22 17:56:25,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:25,347 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:25,357 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#556 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:25,363 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#557 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:25,372 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/69c57dddaf8a46c08ed1bf6bf6931455 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/69c57dddaf8a46c08ed1bf6bf6931455 2023-03-22 17:56:25,373 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322ed14911bfce042e8bacbebbbaec4df0c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ed14911bfce042e8bacbebbbaec4df0c 2023-03-22 17:56:25,377 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into 69c57dddaf8a46c08ed1bf6bf6931455(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:25,377 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:25,377 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=12, startTime=1679507785343; duration=0sec 2023-03-22 17:56:25,377 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:25,378 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:25,379 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:25,379 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:25,379 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:25,379 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7c5931d587cc4cfda78ba6e128321e70, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c94275372931452d862997cabc0ca6f0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/977f19a222f34be69142b2543a86bdf3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/73afeff9f0db44429968c30412653ec2] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=46.9 K 2023-03-22 17:56:25,380 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7c5931d587cc4cfda78ba6e128321e70, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679507781633 2023-03-22 17:56:25,380 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting c94275372931452d862997cabc0ca6f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679507782760 2023-03-22 17:56:25,380 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/e0bb0a167edc48019768c4f128dab058 2023-03-22 17:56:25,381 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 977f19a222f34be69142b2543a86bdf3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679507783422 2023-03-22 17:56:25,381 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 73afeff9f0db44429968c30412653ec2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507784071 2023-03-22 17:56:25,386 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/72f6b9174c4c40f5a78fc1f41f12e669 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/72f6b9174c4c40f5a78fc1f41f12e669 2023-03-22 17:56:25,389 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#559 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:25,393 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into 72f6b9174c4c40f5a78fc1f41f12e669(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:25,393 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:25,393 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=12, startTime=1679507785343; duration=0sec 2023-03-22 17:56:25,393 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:25,401 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/2fbec198b0bd406ea9ae4104cd95cb8e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2fbec198b0bd406ea9ae4104cd95cb8e 2023-03-22 17:56:25,405 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into 2fbec198b0bd406ea9ae4104cd95cb8e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:25,405 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:25,405 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=12, startTime=1679507785346; duration=0sec 2023-03-22 17:56:25,405 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:25,797 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/0649b3b5fe4f402bbcee328089c5aec7 2023-03-22 17:56:26,211 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/952b937dc8fd4f988df2662fd9ec232c 2023-03-22 17:56:26,214 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/e0bb0a167edc48019768c4f128dab058 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e0bb0a167edc48019768c4f128dab058 2023-03-22 17:56:26,218 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e0bb0a167edc48019768c4f128dab058, entries=150, sequenceid=195, filesize=25.3 K 2023-03-22 17:56:26,219 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/0649b3b5fe4f402bbcee328089c5aec7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0649b3b5fe4f402bbcee328089c5aec7 2023-03-22 17:56:26,222 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0649b3b5fe4f402bbcee328089c5aec7, entries=150, sequenceid=195, filesize=11.8 K 2023-03-22 17:56:26,223 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/952b937dc8fd4f988df2662fd9ec232c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/952b937dc8fd4f988df2662fd9ec232c 2023-03-22 17:56:26,226 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/952b937dc8fd4f988df2662fd9ec232c, entries=150, sequenceid=195, filesize=11.8 K 2023-03-22 17:56:26,227 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 876c388f3b400624afff0881af885f88 in 880ms, sequenceid=195, compaction requested=false 2023-03-22 17:56:26,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:26,227 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:26,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:26,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:26,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:26,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,229 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@356be94e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:26,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:26,231 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,233 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:26,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,233 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:26,235 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:26,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,235 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:26,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:26,238 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:26,238 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:26,238 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:26,238 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:26,238 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:26,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,240 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,240 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,244 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:26,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,244 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,245 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,245 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:26,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,245 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:26,245 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:26,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:26,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:26,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:26,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:26,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:26,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:26,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:26,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:26,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507786310, max timestamp=1679507786320Num uniques 859; 2023-03-22 17:56:26,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:26,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507786310, max timestamp=1679507786320Num uniques 859; 2023-03-22 17:56:26,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507786310, max timestamp=1679507786320Num uniques 859; 2023-03-22 17:56:26,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:26,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:26,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507846340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507846341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,345 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:26,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507846343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:26,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:26,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:26,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:26,348 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:26,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:26,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:26,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,349 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,350 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,350 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:26,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:26,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:26,352 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,352 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:26,352 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:26,353 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,356 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:26,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3802a9e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:26,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:26,358 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:26,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:26,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:26,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,360 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:26,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:26,363 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:26,363 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:26,363 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:26,363 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:26,363 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:26,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,364 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,365 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:26,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:26,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,369 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,369 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:26,369 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:26,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507846444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507846444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507846446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,470 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:26,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:26,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:26,473 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:26,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:26,473 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:26,474 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,474 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,475 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,475 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:26,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:26,477 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,477 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:26,478 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:26,478 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:26,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:26,479 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ef3239[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:26,481 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:26,481 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,486 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:26,487 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,487 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:26,488 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:26,491 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:26,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:26,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:26,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:26,491 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:26,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,493 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,493 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,495 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,497 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:26,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,497 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:26,498 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:26,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:26,598 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:26,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:26,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:26,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:26,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:26,601 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:26,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:26,601 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:26,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,602 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,604 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,605 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,605 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:26,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:26,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:26,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,608 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:26,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f055aad[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:26,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:26,610 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,611 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:26,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:26,611 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:26,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,611 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:26,612 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:26,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,612 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:26,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:26,615 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:26,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:26,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:26,615 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:26,615 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:26,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:26,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,623 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:26,623 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:26,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507846648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507846649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507846651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,715 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322feeb8e2531e64e08bcd1bf032231f23a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322feeb8e2531e64e08bcd1bf032231f23a 2023-03-22 17:56:26,719 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/26faa201b43f48a695f73f54f8f9ae1b 2023-03-22 17:56:26,723 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:26,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:26,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:26,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:26,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:26,727 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:26,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:26,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:26,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,727 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,729 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,729 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,729 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:26,729 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:26,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:26,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:26,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:26,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:26,731 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:26,731 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,731 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:26,731 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:26,732 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:26,733 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,733 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,734 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,734 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@154efd9a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:26,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:26,735 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,737 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:26,739 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:26,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:26,742 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:26,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:26,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:26,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:26,742 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:26,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,744 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,745 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,748 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:26,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:26,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,749 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:26,749 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:26,849 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:26,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:26,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:26,852 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:26,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:26,852 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:26,853 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,853 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,855 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,855 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:26,857 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,857 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:26,857 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:26,857 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:26,857 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:26,859 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1798e177[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:26,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:26,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,861 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:26,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,862 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:26,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:26,862 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:26,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:26,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,866 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,866 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:26,868 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:26,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:26,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:26,868 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:26,868 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:26,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,870 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,870 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:26,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:26,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:26,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:26,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,874 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:26,874 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,875 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,875 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:26,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507846952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507846954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:26,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507846957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,975 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:26,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:26,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:26,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:26,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:26,979 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:26,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:26,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:26,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,979 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,980 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,980 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:26,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:26,980 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,980 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:26,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:26,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:26,982 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:26,983 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,983 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:26,983 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:26,983 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:26,983 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:26,985 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46b77747[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:26,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:26,987 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,988 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:26,988 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:26,989 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:26,989 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:26,989 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,989 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:26,997 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:26,999 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:26,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:27,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:27,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:27,000 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:27,002 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,002 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,002 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,003 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,004 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,008 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:27,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:27,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:27,008 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:27,008 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:27,108 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:27,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:27,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:27,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:27,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:27,112 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:27,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:27,112 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:27,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,114 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,114 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,115 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,115 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:27,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:27,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:27,116 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,116 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:27,116 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:27,116 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:27,118 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13b28d0e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:27,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:27,120 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,121 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:27,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:27,121 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:27,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:27,123 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:27,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,123 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:27,125 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:27,125 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:27,125 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:27,125 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:27,125 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:27,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,127 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,127 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/d4d4632900a244afa2b0e364c0f3d9e0 2023-03-22 17:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:27,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:27,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:27,131 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,132 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,132 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,132 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:27,132 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:27,232 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:27,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:27,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:27,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:27,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:27,241 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:27,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:27,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:27,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,243 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,243 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:27,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:27,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:27,245 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,245 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:27,245 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:27,245 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:27,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,248 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:27,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f200cf[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:27,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:27,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,250 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,250 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:27,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:27,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:27,250 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:27,252 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:27,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,252 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:27,254 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:27,254 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:27,254 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:27,254 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:27,254 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:27,256 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,256 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,256 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,257 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,264 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,264 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,264 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:27,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:27,264 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,264 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:27,265 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:27,264 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,264 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,265 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,265 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,265 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,265 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,365 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:27,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:27,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:27,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:27,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:27,368 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:27,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:27,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:27,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,369 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,370 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,370 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:27,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:27,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:27,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,371 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:27,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:27,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:27,372 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,372 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:27,372 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:27,372 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:27,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:27,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:27,372 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:27,373 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,373 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,374 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,374 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:27,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2131d4df[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:27,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:27,375 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,376 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:27,376 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,376 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:27,378 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,378 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:27,378 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,378 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:27,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:27,381 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:27,381 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:27,381 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:27,381 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:27,381 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:27,382 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,382 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,382 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,383 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:27,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,386 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,386 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:27,386 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:27,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:27,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507847457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:27,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507847457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:27,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507847461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,486 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:27,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:27,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:27,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:27,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:27,490 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:27,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:27,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:27,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,491 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,491 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:27,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:27,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:27,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:27,493 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,493 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:27,493 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:27,493 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,493 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:27,494 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:27,496 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,496 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:27,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4182f1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:27,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:27,498 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,500 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:27,500 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:27,500 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,500 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:27,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:27,501 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,501 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:27,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:27,504 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:27,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:27,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:27,504 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:27,504 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:27,505 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,505 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,506 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:27,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:27,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,506 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,507 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,510 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,511 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:27,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:27,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,511 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:27,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:27,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:27,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:27,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:27,511 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:27,511 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:27,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d8c1d93e5d694272ba10799bded5429c 2023-03-22 17:56:27,543 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/26faa201b43f48a695f73f54f8f9ae1b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/26faa201b43f48a695f73f54f8f9ae1b 2023-03-22 17:56:27,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/26faa201b43f48a695f73f54f8f9ae1b, entries=400, sequenceid=209, filesize=59.4 K 2023-03-22 17:56:27,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/d4d4632900a244afa2b0e364c0f3d9e0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d4d4632900a244afa2b0e364c0f3d9e0 2023-03-22 17:56:27,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d4d4632900a244afa2b0e364c0f3d9e0, entries=150, sequenceid=209, filesize=11.8 K 2023-03-22 17:56:27,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d8c1d93e5d694272ba10799bded5429c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d8c1d93e5d694272ba10799bded5429c 2023-03-22 17:56:27,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d8c1d93e5d694272ba10799bded5429c, entries=150, sequenceid=209, filesize=11.8 K 2023-03-22 17:56:27,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 876c388f3b400624afff0881af885f88 in 1262ms, sequenceid=209, compaction requested=true 2023-03-22 17:56:27,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:27,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:27,555 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:27,555 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:27,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:27,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:27,556 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 112920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:27,556 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:27,556 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,556 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/72f6b9174c4c40f5a78fc1f41f12e669, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e0bb0a167edc48019768c4f128dab058, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/26faa201b43f48a695f73f54f8f9ae1b] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=110.3 K 2023-03-22 17:56:27,556 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:27,557 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:27,557 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,557 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/69c57dddaf8a46c08ed1bf6bf6931455, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0649b3b5fe4f402bbcee328089c5aec7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d4d4632900a244afa2b0e364c0f3d9e0] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=35.5 K 2023-03-22 17:56:27,557 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 72f6b9174c4c40f5a78fc1f41f12e669, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507784071 2023-03-22 17:56:27,557 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e0bb0a167edc48019768c4f128dab058, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679507784128 2023-03-22 17:56:27,557 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 69c57dddaf8a46c08ed1bf6bf6931455, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507784071 2023-03-22 17:56:27,557 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 26faa201b43f48a695f73f54f8f9ae1b, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679507786261 2023-03-22 17:56:27,558 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0649b3b5fe4f402bbcee328089c5aec7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679507784128 2023-03-22 17:56:27,558 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d4d4632900a244afa2b0e364c0f3d9e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679507786279 2023-03-22 17:56:27,565 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#565 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:27,569 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#564 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:27,584 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/b99618850fd44a02812ec587ac360ec8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b99618850fd44a02812ec587ac360ec8 2023-03-22 17:56:27,590 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into b99618850fd44a02812ec587ac360ec8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:27,590 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:27,590 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=13, startTime=1679507787555; duration=0sec 2023-03-22 17:56:27,590 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:27,590 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:27,591 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:27,591 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:27,591 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,591 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2fbec198b0bd406ea9ae4104cd95cb8e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/952b937dc8fd4f988df2662fd9ec232c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d8c1d93e5d694272ba10799bded5429c] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=35.5 K 2023-03-22 17:56:27,592 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2fbec198b0bd406ea9ae4104cd95cb8e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679507784071 2023-03-22 17:56:27,592 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 952b937dc8fd4f988df2662fd9ec232c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679507784128 2023-03-22 17:56:27,592 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d8c1d93e5d694272ba10799bded5429c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679507786279 2023-03-22 17:56:27,605 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#566 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:27,611 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:27,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:27,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:27,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:27,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:27,616 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:27,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:27,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:27,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,617 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,617 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/907c9b33e5734c398d7b91d8912f8a6c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/907c9b33e5734c398d7b91d8912f8a6c 2023-03-22 17:56:27,618 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:27,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,619 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/0926c76b8ae7466794f68bc4139005de as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0926c76b8ae7466794f68bc4139005de 2023-03-22 17:56:27,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:27,619 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:27,620 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:27,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:27,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:27,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:27,621 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:27,621 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:27,621 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:27,621 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-22 17:56:27,622 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:27,622 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:27,622 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:27,622 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:27,622 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:27,622 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:27,625 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into 907c9b33e5734c398d7b91d8912f8a6c(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:27,625 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:27,625 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=13, startTime=1679507787555; duration=0sec 2023-03-22 17:56:27,625 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:27,628 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into 0926c76b8ae7466794f68bc4139005de(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:27,628 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:27,628 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=13, startTime=1679507787555; duration=0sec 2023-03-22 17:56:27,628 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:27,645 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322f4c740363a8540bea51bc3b535d68023 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f4c740363a8540bea51bc3b535d68023 2023-03-22 17:56:27,658 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/91489334b4854fa48a26dbd2cd771116 2023-03-22 17:56:27,667 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/4bec829fcbf9460db9c2176859ebafb0 2023-03-22 17:56:28,076 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d5550e42ac924f348c542c34e9c881c7 2023-03-22 17:56:28,080 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/91489334b4854fa48a26dbd2cd771116 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91489334b4854fa48a26dbd2cd771116 2023-03-22 17:56:28,084 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91489334b4854fa48a26dbd2cd771116, entries=150, sequenceid=245, filesize=25.3 K 2023-03-22 17:56:28,084 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/4bec829fcbf9460db9c2176859ebafb0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4bec829fcbf9460db9c2176859ebafb0 2023-03-22 17:56:28,088 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4bec829fcbf9460db9c2176859ebafb0, entries=150, sequenceid=245, filesize=11.8 K 2023-03-22 17:56:28,088 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d5550e42ac924f348c542c34e9c881c7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d5550e42ac924f348c542c34e9c881c7 2023-03-22 17:56:28,093 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d5550e42ac924f348c542c34e9c881c7, entries=150, sequenceid=245, filesize=11.8 K 2023-03-22 17:56:28,093 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 876c388f3b400624afff0881af885f88 in 472ms, sequenceid=245, compaction requested=false 2023-03-22 17:56:28,093 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:28,094 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:28,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:28,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:28,094 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:28,095 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,095 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,096 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:28,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1cbe8a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:28,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:28,097 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,099 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:28,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:28,099 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:28,099 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,099 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:28,101 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:28,103 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:28,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:28,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:28,104 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:28,104 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:28,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,105 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,106 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,107 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,113 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:28,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:28,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,113 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:28,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,113 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:28,113 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,214 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:28,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:28,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:28,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:28,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:28,220 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:28,220 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,220 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:28,220 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:28,224 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,224 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,225 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,225 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:28,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:28,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:28,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:28,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,227 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:28,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:28,229 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:28,229 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,229 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:28,229 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-22 17:56:28,229 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:28,229 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:28,229 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:28,230 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:28,230 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:28,230 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:28,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:28,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:28,257 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303224f8dea7620154323b8704325a2033d22 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303224f8dea7620154323b8704325a2033d22 2023-03-22 17:56:28,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507788258, max timestamp=1679507788265Num uniques 859; 2023-03-22 17:56:28,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507788258, max timestamp=1679507788265Num uniques 859; 2023-03-22 17:56:28,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:28,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:28,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507788258, max timestamp=1679507788265Num uniques 859; 2023-03-22 17:56:28,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:28,275 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/dc2aaf4052ab499dbf9a1ecb9f419e46 2023-03-22 17:56:28,285 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/a02970dad27541019fd65defbc9f749d 2023-03-22 17:56:28,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507788286, max timestamp=1679507788298Num uniques 1068; 2023-03-22 17:56:28,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507788286, max timestamp=1679507788298Num uniques 1068; 2023-03-22 17:56:28,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679507788286, max timestamp=1679507788298Num uniques 1068; 2023-03-22 17:56:28,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-22 17:56:28,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-22 17:56:28,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-22 17:56:28,309 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/f0a2fb49d6b34de0bd8f4824d17efc19 2023-03-22 17:56:28,313 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/dc2aaf4052ab499dbf9a1ecb9f419e46 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/dc2aaf4052ab499dbf9a1ecb9f419e46 2023-03-22 17:56:28,317 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/dc2aaf4052ab499dbf9a1ecb9f419e46, entries=50, sequenceid=251, filesize=11.6 K 2023-03-22 17:56:28,318 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/a02970dad27541019fd65defbc9f749d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/a02970dad27541019fd65defbc9f749d 2023-03-22 17:56:28,324 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/a02970dad27541019fd65defbc9f749d, entries=50, sequenceid=251, filesize=7.1 K 2023-03-22 17:56:28,325 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/f0a2fb49d6b34de0bd8f4824d17efc19 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f0a2fb49d6b34de0bd8f4824d17efc19 2023-03-22 17:56:28,330 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f0a2fb49d6b34de0bd8f4824d17efc19, entries=50, sequenceid=251, filesize=7.1 K 2023-03-22 17:56:28,331 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=114.05 KB/116790 for 876c388f3b400624afff0881af885f88 in 102ms, sequenceid=251, compaction requested=true 2023-03-22 17:56:28,331 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:28,331 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:28,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:28,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=120.76 KB heapSize=339.61 KB 2023-03-22 17:56:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:28,333 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,333 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,333 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,333 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:28,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a89395e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:28,337 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:28,337 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,338 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:28,338 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,338 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:28,340 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:28,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507848349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:28,352 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:28,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:28,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:28,352 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:28,352 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:28,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,354 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,355 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507848352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,357 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:28,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:28,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:28,361 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:28,361 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:28,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,361 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,361 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,373 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322b2eda7d422c246f29620e43acd68d884 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b2eda7d422c246f29620e43acd68d884 2023-03-22 17:56:28,386 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=303, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/b10c2cd8a11f4d6e988935f13d1f390d 2023-03-22 17:56:28,403 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/0395df31b14f46218236ecc27e56a4ce 2023-03-22 17:56:28,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507848453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507848457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,461 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:28,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507848460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:28,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:28,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:28,465 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:28,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:28,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:28,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,466 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,466 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507848465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,467 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:28,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507848465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:28,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:28,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:28,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:28,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:28,477 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:28,478 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,478 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:28,478 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:28,478 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:28,478 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:28,481 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,481 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:28,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@153185fc[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:28,483 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:28,483 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,484 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:28,485 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,485 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:28,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:28,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,486 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,486 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:28,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:28,490 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:28,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:28,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:28,490 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:28,490 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:28,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,492 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,492 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,494 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,497 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,498 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,498 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:28,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:28,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:28,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,498 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:28,498 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:28,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,600 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:28,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:28,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:28,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:28,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:28,604 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:28,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:28,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:28,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,604 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,606 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,606 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:28,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:28,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:28,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:28,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:28,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:28,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:28,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:28,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:28,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:28,608 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:28,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:28,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:28,608 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,610 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:28,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b7dba1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:28,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:28,613 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,614 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,614 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,614 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:28,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:28,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:28,614 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:28,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:28,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:28,621 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:28,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:28,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:28,621 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:28,621 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:28,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,623 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,624 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,626 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,628 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,628 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,628 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:28,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,628 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:28,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:28,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,629 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,629 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:28,629 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:28,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507848661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507848661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,729 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:28,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:28,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:28,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:28,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:28,732 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:28,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:28,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:28,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,733 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,737 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,737 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:28,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:28,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:28,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:28,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:28,739 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:28,740 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,741 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7be1c573[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:28,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:28,742 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,743 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:28,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,743 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:28,745 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:28,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,745 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:28,747 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:28,747 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:28,747 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:28,747 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:28,747 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:28,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:28,750 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:28,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,751 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:28,753 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:28,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:28,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:28,754 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:28,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:28,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:28,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:28,754 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:28,754 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:28,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/0763f78e1976466a81f9cb34d51aab85 2023-03-22 17:56:28,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/b10c2cd8a11f4d6e988935f13d1f390d as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b10c2cd8a11f4d6e988935f13d1f390d 2023-03-22 17:56:28,839 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b10c2cd8a11f4d6e988935f13d1f390d, entries=200, sequenceid=303, filesize=32.3 K 2023-03-22 17:56:28,840 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/0395df31b14f46218236ecc27e56a4ce as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0395df31b14f46218236ecc27e56a4ce 2023-03-22 17:56:28,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0395df31b14f46218236ecc27e56a4ce, entries=150, sequenceid=303, filesize=11.9 K 2023-03-22 17:56:28,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/0763f78e1976466a81f9cb34d51aab85 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0763f78e1976466a81f9cb34d51aab85 2023-03-22 17:56:28,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0763f78e1976466a81f9cb34d51aab85, entries=150, sequenceid=303, filesize=11.9 K 2023-03-22 17:56:28,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~359.48 KB/368112, currentSize=53.67 KB/54960 for 876c388f3b400624afff0881af885f88 in 514ms, sequenceid=303, compaction requested=true 2023-03-22 17:56:28,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:28,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:28,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:28,847 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:28,847 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:28,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:28,848 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97160 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:28,848 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:28,848 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:28,848 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:28,848 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,848 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,848 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/907c9b33e5734c398d7b91d8912f8a6c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91489334b4854fa48a26dbd2cd771116, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/dc2aaf4052ab499dbf9a1ecb9f419e46, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b10c2cd8a11f4d6e988935f13d1f390d] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=94.9 K 2023-03-22 17:56:28,848 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b99618850fd44a02812ec587ac360ec8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4bec829fcbf9460db9c2176859ebafb0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/a02970dad27541019fd65defbc9f749d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0395df31b14f46218236ecc27e56a4ce] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=42.9 K 2023-03-22 17:56:28,848 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 907c9b33e5734c398d7b91d8912f8a6c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679507786279 2023-03-22 17:56:28,849 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b99618850fd44a02812ec587ac360ec8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679507786279 2023-03-22 17:56:28,849 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 91489334b4854fa48a26dbd2cd771116, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679507786335 2023-03-22 17:56:28,849 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4bec829fcbf9460db9c2176859ebafb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679507786335 2023-03-22 17:56:28,850 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting dc2aaf4052ab499dbf9a1ecb9f419e46, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679507788216 2023-03-22 17:56:28,850 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting a02970dad27541019fd65defbc9f749d, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679507788216 2023-03-22 17:56:28,850 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b10c2cd8a11f4d6e988935f13d1f390d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1679507788316 2023-03-22 17:56:28,851 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0395df31b14f46218236ecc27e56a4ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1679507788316 2023-03-22 17:56:28,855 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:28,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:28,859 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#576 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:28,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:28,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:28,859 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:28,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:28,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:28,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,860 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,861 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:28,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:28,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:28,861 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:28,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:28,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:28,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:28,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:28,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:28,864 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:28,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:28,865 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:28,865 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,865 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:28,865 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:56:28,865 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:28,865 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:28,865 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:28,865 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:28,865 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:28,865 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:28,866 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#577 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:28,895 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/dc4587e8b5a1476f93f5217b46252a25 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/dc4587e8b5a1476f93f5217b46252a25 2023-03-22 17:56:28,900 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032285bdd27bb99d42c8a94c20110fbe282f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032285bdd27bb99d42c8a94c20110fbe282f 2023-03-22 17:56:28,918 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into dc4587e8b5a1476f93f5217b46252a25(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:28,918 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:28,918 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=12, startTime=1679507788847; duration=0sec 2023-03-22 17:56:28,918 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:28,918 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:28,920 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:28,920 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:28,920 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:28,920 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0926c76b8ae7466794f68bc4139005de, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d5550e42ac924f348c542c34e9c881c7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f0a2fb49d6b34de0bd8f4824d17efc19, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0763f78e1976466a81f9cb34d51aab85] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=42.9 K 2023-03-22 17:56:28,921 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0926c76b8ae7466794f68bc4139005de, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679507786279 2023-03-22 17:56:28,921 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d5550e42ac924f348c542c34e9c881c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679507786335 2023-03-22 17:56:28,921 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f0a2fb49d6b34de0bd8f4824d17efc19, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679507788216 2023-03-22 17:56:28,922 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 0763f78e1976466a81f9cb34d51aab85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1679507788316 2023-03-22 17:56:28,929 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#579 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:28,930 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=314, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/9ef86991c36a467d9594a4410761b850 2023-03-22 17:56:28,937 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/e58fae62c00f466bbb64a5420bde4ee0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/e58fae62c00f466bbb64a5420bde4ee0 2023-03-22 17:56:28,942 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into e58fae62c00f466bbb64a5420bde4ee0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:28,942 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:28,942 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=12, startTime=1679507788847; duration=0sec 2023-03-22 17:56:28,942 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:28,945 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/b2aebec835a444acbe91fd03b48e7d5a 2023-03-22 17:56:28,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:28,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:29,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507788997, max timestamp=1679507789007Num uniques 859; 2023-03-22 17:56:29,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507788997, max timestamp=1679507789007Num uniques 859; 2023-03-22 17:56:29,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507788997, max timestamp=1679507789007Num uniques 859; 2023-03-22 17:56:29,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:29,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:29,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:29,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507849034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507849035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507849138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507849138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,334 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/b629286ff85c43609d48b41d011a1b60 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b629286ff85c43609d48b41d011a1b60 2023-03-22 17:56:29,338 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into b629286ff85c43609d48b41d011a1b60(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:29,338 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:29,338 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=12, startTime=1679507788847; duration=0sec 2023-03-22 17:56:29,338 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:29,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507849340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507849341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,392 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d9022796386a4f67abdcbbc2f0f12a06 2023-03-22 17:56:29,396 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/9ef86991c36a467d9594a4410761b850 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/9ef86991c36a467d9594a4410761b850 2023-03-22 17:56:29,401 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/9ef86991c36a467d9594a4410761b850, entries=150, sequenceid=314, filesize=25.4 K 2023-03-22 17:56:29,402 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/b2aebec835a444acbe91fd03b48e7d5a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b2aebec835a444acbe91fd03b48e7d5a 2023-03-22 17:56:29,406 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b2aebec835a444acbe91fd03b48e7d5a, entries=150, sequenceid=314, filesize=11.9 K 2023-03-22 17:56:29,406 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d9022796386a4f67abdcbbc2f0f12a06 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d9022796386a4f67abdcbbc2f0f12a06 2023-03-22 17:56:29,411 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d9022796386a4f67abdcbbc2f0f12a06, entries=150, sequenceid=314, filesize=11.9 K 2023-03-22 17:56:29,411 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 876c388f3b400624afff0881af885f88 in 546ms, sequenceid=314, compaction requested=false 2023-03-22 17:56:29,411 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:29,411 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:29,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:29,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:29,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:29,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d14098[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:29,415 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:29,415 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:29,418 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:29,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:29,418 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,418 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:29,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:29,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:29,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:29,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:29,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:29,422 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:29,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:29,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:29,423 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:29,423 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:29,424 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:29,425 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,425 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:29,425 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:29,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,426 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:29,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:29,432 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:29,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:29,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:29,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:29,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:29,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:29,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:29,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,432 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:29,432 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:29,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:29,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:29,433 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:29,533 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:29,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:29,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:29,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:29,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:29,536 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:29,536 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:29,536 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:29,536 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:29,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:29,539 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:29,539 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:29,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:29,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:29,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:29,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:29,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:29,541 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:29,541 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:29,541 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:29,541 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:29,541 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:29,541 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:29,541 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:29,541 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-22 17:56:29,541 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:29,542 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:29,542 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:29,542 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:29,542 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:29,542 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:29,561 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322c6f572f96c784ff3a63a8ed619c6d264 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c6f572f96c784ff3a63a8ed619c6d264 2023-03-22 17:56:29,569 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/6bce472614af408fa6d4073d3af09fae 2023-03-22 17:56:29,579 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/b51dd99604c64a4a994b6d894eef1113 2023-03-22 17:56:29,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:29,647 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:29,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507849658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507849660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507849762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507849763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507849964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:29,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507849966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:29,990 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/9191dee92662487086cc93cdbb44e0f6 2023-03-22 17:56:29,993 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/6bce472614af408fa6d4073d3af09fae as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6bce472614af408fa6d4073d3af09fae 2023-03-22 17:56:29,997 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6bce472614af408fa6d4073d3af09fae, entries=150, sequenceid=352, filesize=25.4 K 2023-03-22 17:56:29,999 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/b51dd99604c64a4a994b6d894eef1113 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b51dd99604c64a4a994b6d894eef1113 2023-03-22 17:56:30,002 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b51dd99604c64a4a994b6d894eef1113, entries=150, sequenceid=352, filesize=11.9 K 2023-03-22 17:56:30,003 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/9191dee92662487086cc93cdbb44e0f6 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9191dee92662487086cc93cdbb44e0f6 2023-03-22 17:56:30,010 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9191dee92662487086cc93cdbb44e0f6, entries=150, sequenceid=352, filesize=11.9 K 2023-03-22 17:56:30,010 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 876c388f3b400624afff0881af885f88 in 469ms, sequenceid=352, compaction requested=true 2023-03-22 17:56:30,011 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:30,011 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:30,012 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,012 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11563a1b[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:30,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:30,014 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,016 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:30,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:30,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:30,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,017 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:30,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:30,020 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:30,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:30,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:30,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:30,020 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:30,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:30,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,024 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,025 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:30,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,028 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:30,028 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:30,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:30,028 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:30,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:30,028 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:30,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:30,129 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:30,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:30,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:30,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:30,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:30,132 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:30,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:30,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:30,132 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:30,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,134 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:30,134 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:30,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,135 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:30,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:30,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:30,139 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,139 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:30,139 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:56:30,139 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:30,139 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:30,139 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:30,139 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:30,139 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:30,139 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:30,152 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322f72684aab28f4351bd8a8d95166182b4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f72684aab28f4351bd8a8d95166182b4 2023-03-22 17:56:30,157 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=363, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/6c8b8ceb2b1c48d084626758b55b8a72 2023-03-22 17:56:30,171 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/d974f0c0da054a079a192c92ea488712 2023-03-22 17:56:30,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:30,270 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507790294, max timestamp=1679507790301Num uniques 859; 2023-03-22 17:56:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507790294, max timestamp=1679507790301Num uniques 859; 2023-03-22 17:56:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507790294, max timestamp=1679507790301Num uniques 859; 2023-03-22 17:56:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:30,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:30,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507850334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507850336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507850439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507850440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507850475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,480 DEBUG [Thread-4009] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:56:30,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507850481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507850481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,488 DEBUG [Thread-4011] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:56:30,489 DEBUG [Thread-4007] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., hostname=jenkins-hbase4.apache.org,34403,1679507617723, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-22 17:56:30,580 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/716e2f5f42714415866fe36970629420 2023-03-22 17:56:30,591 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/6c8b8ceb2b1c48d084626758b55b8a72 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6c8b8ceb2b1c48d084626758b55b8a72 2023-03-22 17:56:30,596 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6c8b8ceb2b1c48d084626758b55b8a72, entries=150, sequenceid=363, filesize=25.4 K 2023-03-22 17:56:30,597 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/d974f0c0da054a079a192c92ea488712 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d974f0c0da054a079a192c92ea488712 2023-03-22 17:56:30,601 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d974f0c0da054a079a192c92ea488712, entries=150, sequenceid=363, filesize=11.9 K 2023-03-22 17:56:30,602 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/716e2f5f42714415866fe36970629420 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/716e2f5f42714415866fe36970629420 2023-03-22 17:56:30,605 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/716e2f5f42714415866fe36970629420, entries=150, sequenceid=363, filesize=11.9 K 2023-03-22 17:56:30,606 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 876c388f3b400624afff0881af885f88 in 467ms, sequenceid=363, compaction requested=true 2023-03-22 17:56:30,606 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:30,606 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:30,606 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:30,607 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:30,615 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:30,616 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,616 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,617 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:30,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a479143[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:30,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:30,618 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,624 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:30,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:30,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:30,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,625 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:30,626 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:30,626 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:30,626 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,626 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:30,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:30,628 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:30,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:30,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:30,628 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:30,628 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:30,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:30,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,630 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:30,631 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,634 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:30,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:30,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:30,634 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:30,634 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:30,634 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:30,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:30,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:30,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:30,645 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-22 17:56:30,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:30,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:30,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:30,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:30,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:30,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:30,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507850664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507850665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,735 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:30,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:30,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:30,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:30,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:30,738 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:30,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:30,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:30,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:30,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,738 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,740 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,741 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:30,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:30,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,741 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:30,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:30,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:30,742 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,742 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:30,742 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:30,742 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:30,742 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:30,744 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,744 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:30,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:30,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff81b25[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:30,746 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:30,746 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,747 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:30,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:30,748 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:30,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:30,749 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:30,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:30,749 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:30,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:30,752 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:30,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:30,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:30,752 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:30,752 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:30,756 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,757 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:30,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,757 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,758 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:30,760 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:30,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:30,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:30,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:30,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:30,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:30,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,760 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:30,760 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:30,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,760 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:30,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507850766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507850769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,861 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:30,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:30,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:30,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:30,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:30,863 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:30,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:30,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:30,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:30,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,864 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,865 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:30,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:30,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,866 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:30,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:30,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:30,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:30,867 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,867 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:30,867 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:30,867 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:30,868 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:30,869 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,869 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,870 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:30,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1daf5dc2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:30,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:30,871 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,873 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:30,873 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,874 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:30,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:30,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:30,875 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,875 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:30,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:30,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:30,878 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:30,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:30,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:30,878 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:30,878 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:30,880 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,880 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,880 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:30,881 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,885 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:30,885 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:30,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:30,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,886 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:30,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:30,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:30,886 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:30,887 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:30,887 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:30,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507850971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1679507850975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,987 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:30,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:30,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:30,990 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:30,990 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:30,990 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:30,990 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:30,990 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,990 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,993 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,993 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:30,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:30,993 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,993 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:30,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:30,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:30,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:30,994 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:30,994 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,994 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:30,995 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:30,995 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:30,995 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:30,997 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:30,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:30,997 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:30,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:30,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d093a8e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:30,998 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:30,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,000 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:31,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:31,006 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:31,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:31,008 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:31,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:31,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:31,008 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:31,008 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:31,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,012 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,013 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,017 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,017 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:31,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:31,018 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:31,018 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:31,018 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,018 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,018 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,018 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,018 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,018 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,018 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,061 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032214691f9e1a7645f983c2e112b49a6746 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032214691f9e1a7645f983c2e112b49a6746 2023-03-22 17:56:31,118 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:31,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:31,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:31,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:31,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:31,121 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:31,121 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,122 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:31,122 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:31,122 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,122 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,125 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,125 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,126 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,126 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:31,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:31,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:31,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:31,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:31,127 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:31,128 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,128 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:31,128 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:31,128 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:31,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:31,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:31,128 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:31,132 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b1279e5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:31,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:31,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,137 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,137 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:31,137 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:31,137 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:31,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,137 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,144 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:31,147 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:31,147 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:31,147 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:31,147 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:31,147 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:31,149 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,149 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,149 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,158 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,158 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,158 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,158 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,158 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,158 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:31,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:31,158 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,159 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,159 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,159 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,159 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:31,159 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:31,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:31,259 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:31,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:31,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:31,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:31,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:31,263 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:31,263 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,263 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:31,263 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:31,263 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,263 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,265 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,265 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,266 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,266 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:31,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:31,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:31,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:31,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:31,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:31,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:31,267 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,267 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:31,267 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:31,267 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:31,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:31,267 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:31,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,269 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:31,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@346c7691[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:31,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:31,271 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,272 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:31,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:31,272 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:31,273 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,273 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:31,274 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:31,274 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,275 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:31,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43486 deadline: 1679507851274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:31,278 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:31,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:31,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:31,278 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:31,278 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:31,279 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,279 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,280 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,281 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:31,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507851279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,282 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,284 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,284 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,285 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:31,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:31,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:31,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,285 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,285 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:31,285 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:31,385 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:31,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:31,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:31,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:31,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:31,389 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:31,389 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,389 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:31,389 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:31,389 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,389 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,390 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,390 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,391 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,391 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:31,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:31,392 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,392 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:31,392 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:31,393 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:31,395 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@534a6c93[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:31,397 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:31,398 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,398 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:31,399 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,399 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:31,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:31,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:31,403 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:31,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:31,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:31,403 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:31,403 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:31,404 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,404 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,404 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,405 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,406 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,413 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:31,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:31,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,413 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,413 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:31,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,467 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=400, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/92f9f9485c174e159bfe814f485e95b4 2023-03-22 17:56:31,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/f2597f9906b1475b80ca034c18beb815 2023-03-22 17:56:31,513 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:31,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:31,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:31,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:31,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:31,522 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:31,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:31,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:31,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,522 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,524 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:31,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:31,525 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:31,525 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,525 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:31,526 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,526 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:31,526 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:31,526 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:31,528 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27e3e13e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:31,529 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:31,529 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,531 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:31,531 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,531 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:31,534 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:31,536 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:31,536 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:31,536 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:31,536 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:31,536 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:31,538 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,538 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,538 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,538 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,539 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,541 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,541 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,541 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,541 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,542 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:31,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:31,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:31,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,542 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:31,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,542 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:31,642 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:31,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:31,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:31,646 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:31,646 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,646 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:31,646 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:31,646 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,646 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,648 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,648 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:31,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:31,649 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:31,649 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,649 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:31,650 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,650 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:31,650 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:31,650 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:31,652 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,652 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,653 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,653 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f467976[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:31,657 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:31,657 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,658 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,658 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:31,658 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:31,658 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:31,658 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,658 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:31,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:31,660 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:31,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:31,662 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:31,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:31,662 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:31,663 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:31,663 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:31,664 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,664 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,664 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,665 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,666 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,669 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,669 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,669 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,669 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:31,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:31,669 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,670 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,670 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,670 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,670 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:31,670 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:31,670 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:31,770 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:31,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:31,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:31,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:31,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:31,773 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:31,773 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,773 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:31,773 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:31,774 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,774 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,775 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,775 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,775 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,776 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:31,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:31,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:31,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:31,776 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:31,776 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,776 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:31,777 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:31,777 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:31,777 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,777 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:31,777 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:31,777 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,777 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:31,777 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:31,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:31,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:31,779 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,779 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,779 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,779 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:31,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@84944f1[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:31,781 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:31,781 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:31,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507851778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,782 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,782 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:31,782 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:31,782 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:31,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:31,785 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:31,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:31,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507851783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:31,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:31,787 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:31,787 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:31,788 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:31,788 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:31,788 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:31,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:31,790 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:31,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,790 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:31,792 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:31,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:31,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:31,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:31,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:31,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:31,799 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:31,799 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:31,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/355ed490c93e4f049de25c873e9c88cf 2023-03-22 17:56:31,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/92f9f9485c174e159bfe814f485e95b4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/92f9f9485c174e159bfe814f485e95b4 2023-03-22 17:56:31,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/92f9f9485c174e159bfe814f485e95b4, entries=200, sequenceid=400, filesize=32.3 K 2023-03-22 17:56:31,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/f2597f9906b1475b80ca034c18beb815 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f2597f9906b1475b80ca034c18beb815 2023-03-22 17:56:31,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f2597f9906b1475b80ca034c18beb815, entries=150, sequenceid=400, filesize=11.9 K 2023-03-22 17:56:31,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/355ed490c93e4f049de25c873e9c88cf as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/355ed490c93e4f049de25c873e9c88cf 2023-03-22 17:56:31,899 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:31,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:31,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:31,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/355ed490c93e4f049de25c873e9c88cf, entries=150, sequenceid=400, filesize=11.9 K 2023-03-22 17:56:31,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 876c388f3b400624afff0881af885f88 in 1257ms, sequenceid=400, compaction requested=true 2023-03-22 17:56:31,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:31,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:31,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:31,902 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:31,902 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:31,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-22 17:56:31,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:31,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:31,902 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:31,902 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,902 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:31,902 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:31,903 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,903 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,904 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:31,904 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:31,904 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,904 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b629286ff85c43609d48b41d011a1b60, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/9ef86991c36a467d9594a4410761b850, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6bce472614af408fa6d4073d3af09fae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6c8b8ceb2b1c48d084626758b55b8a72, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/92f9f9485c174e159bfe814f485e95b4] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=134.6 K 2023-03-22 17:56:31,904 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:31,904 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:31,904 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,904 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/dc4587e8b5a1476f93f5217b46252a25, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b2aebec835a444acbe91fd03b48e7d5a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b51dd99604c64a4a994b6d894eef1113, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d974f0c0da054a079a192c92ea488712, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f2597f9906b1475b80ca034c18beb815] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=60.0 K 2023-03-22 17:56:31,905 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b629286ff85c43609d48b41d011a1b60, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1679507788316 2023-03-22 17:56:31,905 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting dc4587e8b5a1476f93f5217b46252a25, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1679507788316 2023-03-22 17:56:31,905 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9ef86991c36a467d9594a4410761b850, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679507788336 2023-03-22 17:56:31,905 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b2aebec835a444acbe91fd03b48e7d5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679507788336 2023-03-22 17:56:31,905 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:31,905 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:31,905 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6bce472614af408fa6d4073d3af09fae, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679507789014 2023-03-22 17:56:31,905 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b51dd99604c64a4a994b6d894eef1113, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679507789014 2023-03-22 17:56:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:31,906 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:31,906 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d974f0c0da054a079a192c92ea488712, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1679507789652 2023-03-22 17:56:31,906 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6c8b8ceb2b1c48d084626758b55b8a72, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1679507789652 2023-03-22 17:56:31,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:31,907 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting f2597f9906b1475b80ca034c18beb815, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679507790334 2023-03-22 17:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:31,907 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 92f9f9485c174e159bfe814f485e95b4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679507790313 2023-03-22 17:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:31,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:31,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:31,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:31,907 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:31,908 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,908 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:31,908 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-22 17:56:31,908 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:31,908 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:31,909 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:31,909 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:31,909 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:31,909 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:31,917 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#592 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:31,930 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#593 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:31,934 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/fad46194dca44bbbb7d3a6a6e0425807 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/fad46194dca44bbbb7d3a6a6e0425807 2023-03-22 17:56:31,938 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into fad46194dca44bbbb7d3a6a6e0425807(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:31,938 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:31,938 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=11, startTime=1679507791902; duration=0sec 2023-03-22 17:56:31,938 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:31,938 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:31,939 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:31,939 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:31,939 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:31,939 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/e58fae62c00f466bbb64a5420bde4ee0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d9022796386a4f67abdcbbc2f0f12a06, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9191dee92662487086cc93cdbb44e0f6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/716e2f5f42714415866fe36970629420, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/355ed490c93e4f049de25c873e9c88cf] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=60.0 K 2023-03-22 17:56:31,940 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e58fae62c00f466bbb64a5420bde4ee0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1679507788316 2023-03-22 17:56:31,940 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting d9022796386a4f67abdcbbc2f0f12a06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679507788336 2023-03-22 17:56:31,940 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 9191dee92662487086cc93cdbb44e0f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679507789014 2023-03-22 17:56:31,940 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 716e2f5f42714415866fe36970629420, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1679507789652 2023-03-22 17:56:31,941 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 355ed490c93e4f049de25c873e9c88cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679507790334 2023-03-22 17:56:31,946 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#594 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:31,952 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/9abb743892124ebfa062d4c88c7e00e7 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9abb743892124ebfa062d4c88c7e00e7 2023-03-22 17:56:31,956 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into 9abb743892124ebfa062d4c88c7e00e7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:31,956 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:31,956 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=11, startTime=1679507791902; duration=0sec 2023-03-22 17:56:31,956 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:31,973 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/7b66fbb8884140789b8d8ed97ae7ed8a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/7b66fbb8884140789b8d8ed97ae7ed8a 2023-03-22 17:56:31,977 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into 7b66fbb8884140789b8d8ed97ae7ed8a(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:31,977 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:31,977 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=11, startTime=1679507791902; duration=0sec 2023-03-22 17:56:31,977 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:32,334 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322a28cb2509e5f4b1f99e2f4b9015cc2c8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a28cb2509e5f4b1f99e2f4b9015cc2c8 2023-03-22 17:56:32,739 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=409, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/15670c3aa4464d5e8bfae7cb5d8d6657 2023-03-22 17:56:32,748 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/4a64ce98c5ed4b01842e5e178321c196 2023-03-22 17:56:32,757 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/654d8c0177534374a6607bf2db7f4665 2023-03-22 17:56:32,760 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/15670c3aa4464d5e8bfae7cb5d8d6657 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/15670c3aa4464d5e8bfae7cb5d8d6657 2023-03-22 17:56:32,764 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/15670c3aa4464d5e8bfae7cb5d8d6657, entries=150, sequenceid=409, filesize=25.4 K 2023-03-22 17:56:32,765 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/4a64ce98c5ed4b01842e5e178321c196 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4a64ce98c5ed4b01842e5e178321c196 2023-03-22 17:56:32,768 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4a64ce98c5ed4b01842e5e178321c196, entries=150, sequenceid=409, filesize=11.9 K 2023-03-22 17:56:32,769 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/654d8c0177534374a6607bf2db7f4665 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/654d8c0177534374a6607bf2db7f4665 2023-03-22 17:56:32,772 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/654d8c0177534374a6607bf2db7f4665, entries=150, sequenceid=409, filesize=11.9 K 2023-03-22 17:56:32,773 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 876c388f3b400624afff0881af885f88 in 865ms, sequenceid=409, compaction requested=false 2023-03-22 17:56:32,773 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:32,773 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:32,775 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:32,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:32,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8e79f9f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:32,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:32,777 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,778 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:32,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:32,778 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:32,778 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,778 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:32,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:32,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:32,780 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,780 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:32,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:32,783 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:32,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:32,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:32,783 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:32,783 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:32,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:32,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,787 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:32,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:32,788 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:32,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:32,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:32,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:32,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:32,792 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:32,793 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:32,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:32,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:32,792 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:32,793 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:32,793 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,793 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,793 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,793 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:32,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:32,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:32,831 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:32,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:32,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:32,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:32,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:32,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:32,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:32,842 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322efad86061fdf441194a21a795d436ef2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322efad86061fdf441194a21a795d436ef2 2023-03-22 17:56:32,848 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=422, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/59643730766a4f378c16015780f85e22 2023-03-22 17:56:32,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/3732390755a04a9f81a2f454da964cb4 2023-03-22 17:56:32,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/38aa6899f02b4eb793ebbcc9417c0478 2023-03-22 17:56:32,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507792854, max timestamp=1679507792866Num uniques 859; 2023-03-22 17:56:32,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507792854, max timestamp=1679507792866Num uniques 859; 2023-03-22 17:56:32,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507792854, max timestamp=1679507792866Num uniques 859; 2023-03-22 17:56:32,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:32,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:32,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:32,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/59643730766a4f378c16015780f85e22 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/59643730766a4f378c16015780f85e22 2023-03-22 17:56:32,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/59643730766a4f378c16015780f85e22, entries=250, sequenceid=422, filesize=39.2 K 2023-03-22 17:56:32,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/3732390755a04a9f81a2f454da964cb4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3732390755a04a9f81a2f454da964cb4 2023-03-22 17:56:32,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3732390755a04a9f81a2f454da964cb4, entries=150, sequenceid=422, filesize=11.9 K 2023-03-22 17:56:32,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/38aa6899f02b4eb793ebbcc9417c0478 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/38aa6899f02b4eb793ebbcc9417c0478 2023-03-22 17:56:32,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/38aa6899f02b4eb793ebbcc9417c0478, entries=150, sequenceid=422, filesize=11.9 K 2023-03-22 17:56:32,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=80.51 KB/82440 for 876c388f3b400624afff0881af885f88 in 52ms, sequenceid=422, compaction requested=true 2023-03-22 17:56:32,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:32,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:32,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:32,884 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:32,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:32,884 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:32,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:32,885 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=87.22 KB heapSize=240.84 KB 2023-03-22 17:56:32,885 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:32,885 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92904 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:32,885 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:32,885 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:32,885 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:32,885 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:32,885 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:32,885 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:32,885 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:32,885 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:32,885 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/7b66fbb8884140789b8d8ed97ae7ed8a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/15670c3aa4464d5e8bfae7cb5d8d6657, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/59643730766a4f378c16015780f85e22] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=90.7 K 2023-03-22 17:56:32,885 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:32,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:32,885 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/fad46194dca44bbbb7d3a6a6e0425807, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4a64ce98c5ed4b01842e5e178321c196, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3732390755a04a9f81a2f454da964cb4] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=36.4 K 2023-03-22 17:56:32,886 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 7b66fbb8884140789b8d8ed97ae7ed8a, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679507790334 2023-03-22 17:56:32,886 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 15670c3aa4464d5e8bfae7cb5d8d6657, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679507790651 2023-03-22 17:56:32,886 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting fad46194dca44bbbb7d3a6a6e0425807, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679507790334 2023-03-22 17:56:32,887 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 59643730766a4f378c16015780f85e22, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679507792809 2023-03-22 17:56:32,887 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 4a64ce98c5ed4b01842e5e178321c196, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679507790651 2023-03-22 17:56:32,888 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3732390755a04a9f81a2f454da964cb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679507792809 2023-03-22 17:56:32,893 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:32,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:32,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:32,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:32,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:32,908 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:32,908 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:32,908 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:32,908 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:32,908 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,909 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:32,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,910 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:32,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:32,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:32,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:32,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:32,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:32,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:32,912 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#601 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:32,912 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:32,912 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:32,913 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:32,913 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:32,913 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:32,913 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:32,913 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,913 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:32,913 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:32,913 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#602 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,915 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:32,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,915 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:32,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507852911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:32,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:32,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,917 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032245ce12b1caaf418087e6b5f915e82343 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032245ce12b1caaf418087e6b5f915e82343 2023-03-22 17:56:32,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:32,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507852915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:32,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:32,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b575ac[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:32,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:32,927 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,930 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:32,930 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:32,930 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:32,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:32,931 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:32,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:32,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:32,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:32,935 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:32,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:32,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:32,936 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:32,936 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:32,937 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,937 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,937 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:32,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,938 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,941 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:32,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:32,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:32,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:32,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:32,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:32,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:32,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:32,945 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:32,945 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:32,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:32,944 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,945 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:32,945 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:32,944 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:32,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:32,947 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=451, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/3b0fa68b31dc4848a5b99f42745b2ce5 2023-03-22 17:56:33,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507853016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507853020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,045 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:33,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:33,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:33,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:33,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:33,048 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:33,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:33,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:33,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,051 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,051 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:33,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:33,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:33,053 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,053 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:33,053 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:33,053 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,054 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:33,054 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:33,054 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,054 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:33,054 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:33,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,055 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:33,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:33,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cd43705[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:33,057 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:33,057 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:33,059 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,059 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:33,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:33,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:33,060 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:33,063 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:33,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:33,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:33,063 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:33,063 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:33,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,065 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,066 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,066 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,070 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,071 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:33,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:33,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,071 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:33,071 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:33,071 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:33,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,171 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:33,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:33,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:33,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:33,175 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:33,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:33,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:33,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:33,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:33,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:33,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:33,183 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,183 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:33,183 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:33,183 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:33,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:33,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:33,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:33,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:33,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef88ea2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:33,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:33,187 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,189 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:33,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:33,189 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,189 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:33,189 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:33,190 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:33,190 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,191 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:33,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:33,194 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:33,194 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:33,194 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:33,194 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:33,194 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:33,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,196 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,198 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,200 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:33,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:33,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:33,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,201 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:33,201 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:33,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,201 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507853218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507853221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,301 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:33,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:33,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:33,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:33,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:33,304 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:33,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:33,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:33,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:33,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:33,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:33,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:33,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:33,309 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,309 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:33,309 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:33,309 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:33,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:33,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,310 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:33,310 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:33,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,313 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:33,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a57e185[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:33,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:33,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:33,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:33,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:33,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:33,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:33,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,320 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,320 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:33,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:33,323 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:33,323 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:33,323 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:33,323 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:33,323 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:33,325 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,325 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,325 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,326 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,328 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:33,331 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/db569ce875514055bebd655a411fb23b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/db569ce875514055bebd655a411fb23b 2023-03-22 17:56:33,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,331 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,331 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:33,332 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:33,331 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,335 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into db569ce875514055bebd655a411fb23b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:33,335 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:33,335 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=13, startTime=1679507792884; duration=0sec 2023-03-22 17:56:33,335 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:33,335 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:33,336 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:33,336 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:33,336 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,336 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9abb743892124ebfa062d4c88c7e00e7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/654d8c0177534374a6607bf2db7f4665, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/38aa6899f02b4eb793ebbcc9417c0478] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=36.4 K 2023-03-22 17:56:33,337 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9abb743892124ebfa062d4c88c7e00e7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679507790334 2023-03-22 17:56:33,337 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 654d8c0177534374a6607bf2db7f4665, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1679507790651 2023-03-22 17:56:33,337 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 38aa6899f02b4eb793ebbcc9417c0478, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679507792809 2023-03-22 17:56:33,347 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#604 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:33,354 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/37f9715918654781a203c97a7206f0b8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/37f9715918654781a203c97a7206f0b8 2023-03-22 17:56:33,357 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into 37f9715918654781a203c97a7206f0b8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:33,357 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:33,357 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=13, startTime=1679507792884; duration=0sec 2023-03-22 17:56:33,357 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:33,357 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/e2f459d776664e51b07c6b60799381ba 2023-03-22 17:56:33,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/6338c55cac46402fb33ed0a2536bdf25 2023-03-22 17:56:33,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/3b0fa68b31dc4848a5b99f42745b2ce5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/3b0fa68b31dc4848a5b99f42745b2ce5 2023-03-22 17:56:33,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/3b0fa68b31dc4848a5b99f42745b2ce5, entries=150, sequenceid=451, filesize=25.4 K 2023-03-22 17:56:33,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/e2f459d776664e51b07c6b60799381ba as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/e2f459d776664e51b07c6b60799381ba 2023-03-22 17:56:33,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/e2f459d776664e51b07c6b60799381ba, entries=150, sequenceid=451, filesize=11.9 K 2023-03-22 17:56:33,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/6338c55cac46402fb33ed0a2536bdf25 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6338c55cac46402fb33ed0a2536bdf25 2023-03-22 17:56:33,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6338c55cac46402fb33ed0a2536bdf25, entries=150, sequenceid=451, filesize=11.9 K 2023-03-22 17:56:33,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~240.80 KB/246576, currentSize=93.93 KB/96180 for 876c388f3b400624afff0881af885f88 in 491ms, sequenceid=451, compaction requested=false 2023-03-22 17:56:33,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:33,432 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:33,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:33,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:33,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:33,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:33,435 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:33,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:33,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:33,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,435 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,437 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,437 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,438 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,438 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:33,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:33,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:33,439 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:33,439 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,439 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:33,440 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-22 17:56:33,440 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:33,440 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:33,440 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:33,440 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:33,440 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:33,440 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:33,454 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322fb113555e2e341859d9ae532427af8db to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322fb113555e2e341859d9ae532427af8db 2023-03-22 17:56:33,460 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=470, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/aa70f8cb913f483ea3648a2f1ce4925f 2023-03-22 17:56:33,469 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/96846c7d73e54017ab9edd923e8836c2 2023-03-22 17:56:33,480 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/11209c658e7742a5862687fb065ef98b 2023-03-22 17:56:33,483 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/aa70f8cb913f483ea3648a2f1ce4925f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/aa70f8cb913f483ea3648a2f1ce4925f 2023-03-22 17:56:33,487 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/aa70f8cb913f483ea3648a2f1ce4925f, entries=150, sequenceid=470, filesize=25.4 K 2023-03-22 17:56:33,488 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/96846c7d73e54017ab9edd923e8836c2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/96846c7d73e54017ab9edd923e8836c2 2023-03-22 17:56:33,490 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/96846c7d73e54017ab9edd923e8836c2, entries=150, sequenceid=470, filesize=11.9 K 2023-03-22 17:56:33,491 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/11209c658e7742a5862687fb065ef98b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/11209c658e7742a5862687fb065ef98b 2023-03-22 17:56:33,494 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/11209c658e7742a5862687fb065ef98b, entries=150, sequenceid=470, filesize=11.9 K 2023-03-22 17:56:33,495 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 876c388f3b400624afff0881af885f88 in 55ms, sequenceid=470, compaction requested=true 2023-03-22 17:56:33,495 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:33,495 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:33,497 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,497 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,498 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:33,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31d4e976[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:33,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:33,499 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,502 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:33,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:33,502 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,502 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:33,502 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:33,504 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,504 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:33,504 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,504 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:33,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:33,506 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:33,507 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:33,507 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:33,507 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:33,507 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:33,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,508 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,509 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,510 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,512 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,512 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,512 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,512 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,512 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:33,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,513 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,513 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,513 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,513 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:33,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,513 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:33,513 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,513 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:33,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:33,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:33,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:33,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:33,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:33,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:33,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:33,576 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303228244694645ef48b49aabcd57b89b6b1b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303228244694645ef48b49aabcd57b89b6b1b 2023-03-22 17:56:33,613 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:33,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:33,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:33,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:33,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:33,616 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:33,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:33,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:33,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,616 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,622 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,622 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:33,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:33,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:33,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:33,624 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:33,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507793601, max timestamp=1679507793624Num uniques 859; 2023-03-22 17:56:33,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507793601, max timestamp=1679507793624Num uniques 859; 2023-03-22 17:56:33,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:33,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:33,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507793601, max timestamp=1679507793624Num uniques 859; 2023-03-22 17:56:33,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:33,628 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,629 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,629 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cc58c56[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:33,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:33,630 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,632 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:33,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:33,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,632 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:33,632 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:33,635 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:33,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:33,637 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:33,637 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:33,637 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:33,637 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:33,637 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:33,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,639 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,640 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,641 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,644 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:33,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,644 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:33,644 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,644 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:33,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507853655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507853657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,744 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:33,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:33,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:33,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:33,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:33,748 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:33,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:33,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:33,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,748 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,749 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,749 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,750 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,750 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:33,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:33,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:33,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:33,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:33,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:33,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:33,751 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,751 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:33,751 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:33,751 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:33,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:33,751 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,752 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:33,752 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:33,754 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,754 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,755 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/17c3a3bf1eb64ca29cb1a62f9e3bcd0a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/17c3a3bf1eb64ca29cb1a62f9e3bcd0a 2023-03-22 17:56:33,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,755 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:33,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@161286f2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:33,757 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:33,757 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,759 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,759 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:33,759 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:33,759 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:33,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,759 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:33,760 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into 17c3a3bf1eb64ca29cb1a62f9e3bcd0a(size=26.2 K), total size for store is 77.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:33,760 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:33,760 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=13, startTime=1679507792884; duration=0sec 2023-03-22 17:56:33,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507853760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:33,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,761 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:33,760 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:33,761 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:33,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:33,764 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:33,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:33,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:33,764 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:33,764 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:33,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507853761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,766 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,766 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,766 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,767 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,768 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,771 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:33,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:33,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,771 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,772 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:33,772 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:33,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,872 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:33,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:33,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:33,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:33,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:33,875 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:33,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:33,875 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:33,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,876 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,877 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,877 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:33,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:33,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:33,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:33,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:33,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:33,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,878 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:33,879 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,879 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:33,879 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:33,879 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:33,881 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,881 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:33,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dc067c[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:33,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:33,883 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,884 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:33,884 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:33,884 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,884 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,887 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:33,889 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:33,889 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:33,889 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:33,889 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:33,889 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:33,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:33,891 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:33,891 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,893 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:33,895 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:33,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:33,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:33,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:33,895 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:33,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:33,895 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:33,895 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:33,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507853963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:33,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507853966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:33,996 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:33,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:33,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:33,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:33,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:33,999 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:33,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:33,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:33,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:33,999 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,000 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,001 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,001 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,002 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:34,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:34,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:34,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:34,003 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,003 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:34,003 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:34,003 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:34,004 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:34,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,008 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=480, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/1f8c2c666beb45abb69e8c846dec92dd 2023-03-22 17:56:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:34,009 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:34,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1555c028[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:34,009 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:34,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:34,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:34,011 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,011 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:34,015 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:34,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:34,020 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:34,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:34,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:34,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:34,020 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,023 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,024 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,025 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,029 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,029 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:34,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:34,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,030 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:34,030 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:34,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/2fe698ed6ed04e2c913adb0ad374e785 2023-03-22 17:56:34,130 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:34,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:34,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:34,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:34,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:34,133 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:34,133 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,134 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:34,134 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:34,134 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,134 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,135 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,136 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:34,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:34,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:34,137 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:34,137 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,137 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:34,137 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:34,137 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:34,137 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,137 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:34,137 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:34,137 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:34,138 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:34,139 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,139 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:34,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@751259e8[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:34,141 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:34,141 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,143 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:34,143 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,144 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:34,145 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:34,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:34,147 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:34,147 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:34,147 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:34,147 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:34,147 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:34,149 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,149 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,149 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,150 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,151 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,154 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,154 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:34,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:34,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:34,155 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:34,155 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:34,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,255 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:34,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:34,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:34,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:34,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:34,258 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:34,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:34,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:34,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,261 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,261 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:34,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:34,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:34,262 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:34,262 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:34,263 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,263 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:34,263 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:34,263 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,265 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773574ae[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:34,267 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:34,267 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,268 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:34,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:34,268 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:34,268 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,268 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:34,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507854266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,269 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:34,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,269 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:34,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:34,272 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:34,272 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:34,272 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:34,272 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:34,272 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:34,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507854271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,275 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,275 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,277 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,282 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,283 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:34,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:34,283 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,283 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:34,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,283 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,283 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:34,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,383 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:34,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:34,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:34,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:34,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:34,387 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:34,387 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,387 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:34,387 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:34,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,396 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,396 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:34,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:34,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:34,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:34,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:34,397 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:34,402 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,402 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,402 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,402 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:34,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@143963d2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:34,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:34,404 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,406 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:34,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:34,406 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:34,407 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,407 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:34,408 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:34,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,408 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:34,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:34,410 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:34,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:34,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:34,410 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:34,410 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:34,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,415 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,416 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,417 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,419 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:34,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:34,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,420 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:34,420 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:34,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,420 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,420 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:34,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/f5aa7f9889e84dc198c702de37eec345 2023-03-22 17:56:34,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/1f8c2c666beb45abb69e8c846dec92dd as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1f8c2c666beb45abb69e8c846dec92dd 2023-03-22 17:56:34,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1f8c2c666beb45abb69e8c846dec92dd, entries=150, sequenceid=480, filesize=25.4 K 2023-03-22 17:56:34,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/2fe698ed6ed04e2c913adb0ad374e785 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2fe698ed6ed04e2c913adb0ad374e785 2023-03-22 17:56:34,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2fe698ed6ed04e2c913adb0ad374e785, entries=100, sequenceid=480, filesize=9.5 K 2023-03-22 17:56:34,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/f5aa7f9889e84dc198c702de37eec345 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f5aa7f9889e84dc198c702de37eec345 2023-03-22 17:56:34,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f5aa7f9889e84dc198c702de37eec345, entries=100, sequenceid=480, filesize=9.5 K 2023-03-22 17:56:34,454 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for 876c388f3b400624afff0881af885f88 in 888ms, sequenceid=480, compaction requested=true 2023-03-22 17:56:34,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:34,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:34,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:34,455 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:34,455 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:34,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:34,456 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:34,456 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104964 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:34,456 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:34,456 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:34,456 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,456 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,456 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/db569ce875514055bebd655a411fb23b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/e2f459d776664e51b07c6b60799381ba, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/96846c7d73e54017ab9edd923e8836c2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2fe698ed6ed04e2c913adb0ad374e785] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=46.0 K 2023-03-22 17:56:34,456 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/17c3a3bf1eb64ca29cb1a62f9e3bcd0a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/3b0fa68b31dc4848a5b99f42745b2ce5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/aa70f8cb913f483ea3648a2f1ce4925f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1f8c2c666beb45abb69e8c846dec92dd] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=102.5 K 2023-03-22 17:56:34,457 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting db569ce875514055bebd655a411fb23b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679507792809 2023-03-22 17:56:34,457 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 17c3a3bf1eb64ca29cb1a62f9e3bcd0a, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679507792809 2023-03-22 17:56:34,458 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting e2f459d776664e51b07c6b60799381ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679507792870 2023-03-22 17:56:34,458 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 3b0fa68b31dc4848a5b99f42745b2ce5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679507792870 2023-03-22 17:56:34,458 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 96846c7d73e54017ab9edd923e8836c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1679507792900 2023-03-22 17:56:34,458 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting aa70f8cb913f483ea3648a2f1ce4925f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1679507792900 2023-03-22 17:56:34,459 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2fe698ed6ed04e2c913adb0ad374e785, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1679507793564 2023-03-22 17:56:34,459 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1f8c2c666beb45abb69e8c846dec92dd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1679507793563 2023-03-22 17:56:34,465 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#612 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:34,469 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#613 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:34,507 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-03-22 17:56:34,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:34,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:34,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:34,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:34,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:34,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:34,520 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:34,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:34,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:34,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:34,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:34,524 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:34,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:34,524 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:34,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,525 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,525 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507854522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507854522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,526 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,526 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,527 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,527 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:34,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507854525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:34,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:34,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:34,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:34,528 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,528 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:34,528 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:34,528 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:34,529 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:34,530 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,531 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:34,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fd6b5fd[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:34,533 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:34,533 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,534 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:34,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:34,537 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:34,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:34,540 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:34,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:34,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:34,540 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:34,540 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:34,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,542 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,545 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:34,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:34,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,548 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:34,548 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:34,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:34,548 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507854626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507854626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507854629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,648 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:34,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:34,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:34,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:34,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:34,652 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:34,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:34,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:34,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,652 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,654 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,654 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:34,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:34,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:34,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:34,655 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:34,655 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,655 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:34,656 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,656 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:34,656 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:34,656 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:34,659 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,659 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,660 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,660 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:34,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486c64f6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:34,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:34,661 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,663 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:34,663 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,663 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:34,665 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:34,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,665 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:34,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:34,667 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:34,667 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:34,667 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:34,667 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:34,668 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:34,670 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,670 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,670 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,672 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,675 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,676 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:34,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:34,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:34,676 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:34,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,676 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:34,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507854771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,776 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:34,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:34,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507854776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:34,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:34,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:34,780 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:34,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:34,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:34,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,780 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,782 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,782 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:34,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:34,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:34,783 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:34,783 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,783 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:34,784 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,784 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:34,784 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:34,784 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:34,785 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,786 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14a0e951[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:34,787 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:34,787 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,789 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:34,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,789 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:34,791 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,791 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:34,791 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,791 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:34,794 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:34,794 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:34,794 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:34,794 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:34,794 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:34,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,796 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,803 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:34,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:34,804 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:34,804 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:34,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507854829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507854830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:34,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1679507854830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,875 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/b85c4cf60982458f8c742411e651b7de as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b85c4cf60982458f8c742411e651b7de 2023-03-22 17:56:34,880 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into b85c4cf60982458f8c742411e651b7de(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:34,880 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:34,880 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=12, startTime=1679507794455; duration=0sec 2023-03-22 17:56:34,881 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:34,881 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:34,882 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:34,882 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:34,882 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,882 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/37f9715918654781a203c97a7206f0b8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6338c55cac46402fb33ed0a2536bdf25, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/11209c658e7742a5862687fb065ef98b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f5aa7f9889e84dc198c702de37eec345] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=46.0 K 2023-03-22 17:56:34,882 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 37f9715918654781a203c97a7206f0b8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679507792809 2023-03-22 17:56:34,883 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6338c55cac46402fb33ed0a2536bdf25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679507792870 2023-03-22 17:56:34,883 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 11209c658e7742a5862687fb065ef98b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1679507792900 2023-03-22 17:56:34,884 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f5aa7f9889e84dc198c702de37eec345, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1679507793564 2023-03-22 17:56:34,891 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#615 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:34,898 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/2dc288712eea48c08af07e1d1f94403c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2dc288712eea48c08af07e1d1f94403c 2023-03-22 17:56:34,902 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into 2dc288712eea48c08af07e1d1f94403c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:34,902 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:34,902 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=12, startTime=1679507794455; duration=0sec 2023-03-22 17:56:34,902 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:34,904 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:34,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:34,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:34,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:34,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:34,907 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:34,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:34,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:34,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,907 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,910 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,910 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:34,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:34,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:34,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,914 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:34,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b582fe4[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:34,917 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:34,917 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,918 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322039b894867b0482ab25eb4b31e221f5a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322039b894867b0482ab25eb4b31e221f5a 2023-03-22 17:56:34,919 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,919 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,919 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:34,919 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:34,919 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:34,920 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:34,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:34,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,922 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,922 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:34,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:34,925 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:34,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:34,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:34,925 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:34,925 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:34,928 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:34,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,929 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:34,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:34,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:34,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,935 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:34,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:34,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:34,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:34,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:34,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:34,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:34,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:34,936 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:34,936 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:34,936 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:34,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:34,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,036 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:35,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:35,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:35,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:35,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:35,039 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:35,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,039 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:35,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:35,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,040 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,041 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,041 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,042 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:35,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:35,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:35,043 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,043 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:35,043 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:35,043 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:35,045 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,045 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,045 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,045 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:35,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513b022f[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:35,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:35,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:35,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:35,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:35,050 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:35,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:35,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:35,052 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:35,052 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:35,053 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:35,053 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:35,053 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:35,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,054 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,055 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,058 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:35,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,058 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:35,059 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:35,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,059 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:35,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507855133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507855134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507855134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,159 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:35,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:35,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:35,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:35,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:35,162 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:35,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:35,162 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:35,163 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,163 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,168 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,168 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:35,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:35,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:35,169 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:35,169 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,169 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:35,170 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,170 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:35,170 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:35,170 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:35,171 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,171 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,172 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,172 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:35,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adb972a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:35,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:35,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,174 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,174 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:35,174 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:35,174 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:35,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,174 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:35,175 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,175 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:35,176 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,176 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:35,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:35,178 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:35,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:35,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:35,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:35,178 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:35,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,180 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,181 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,184 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,184 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:35,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:35,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:35,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,185 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:35,185 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:35,285 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:35,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:35,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:35,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:35,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:35,291 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:35,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:35,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:35,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,291 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,293 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/4b0f2adbc4364909aee8f408cd648fa9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4b0f2adbc4364909aee8f408cd648fa9 2023-03-22 17:56:35,297 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,297 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:35,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:35,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:35,299 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:35,299 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,299 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:35,300 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:35,300 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:35,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:35,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:35,300 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:35,301 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into 4b0f2adbc4364909aee8f408cd648fa9(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:35,301 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:35,301 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=12, startTime=1679507794455; duration=0sec 2023-03-22 17:56:35,301 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:35,302 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,302 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,303 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,303 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:35,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a20341[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:35,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:35,304 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,305 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,306 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:35,306 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:35,306 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:35,306 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,306 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:35,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,307 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:35,307 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,307 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:35,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:35,310 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:35,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:35,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:35,310 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:35,310 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:35,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,312 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,312 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,314 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:35,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:35,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,317 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:35,318 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:35,317 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:35,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,318 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,326 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=518, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/c1bf165b419943ecbca9be3b51bd7b7a 2023-03-22 17:56:35,418 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:35,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:35,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:35,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:35,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:35,421 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:35,421 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,421 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:35,421 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:35,421 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,421 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,422 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,422 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,422 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,422 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:35,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:35,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:35,424 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,424 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:35,424 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:35,424 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:35,425 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,425 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,426 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:35,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@630f269e[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:35,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:35,427 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,428 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,428 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,428 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:35,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:35,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:35,429 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:35,432 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,432 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:35,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:35,434 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:35,434 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:35,434 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:35,434 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:35,434 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:35,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,436 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,437 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,439 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:35,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:35,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:35,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,440 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,440 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:35,440 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:35,540 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:35,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:35,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:35,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:35,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:35,544 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:35,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:35,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:35,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,544 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,546 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,546 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,546 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,547 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:35,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:35,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:35,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:35,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:35,548 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,548 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:35,549 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:35,550 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,550 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,551 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aac6aaa[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:35,553 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:35,553 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,554 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:35,554 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:35,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,555 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:35,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:35,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,556 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:35,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:35,558 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:35,558 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:35,558 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:35,558 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:35,558 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:35,560 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,560 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,560 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,561 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,562 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:35,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,565 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,565 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,565 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:35,566 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:35,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43456 deadline: 1679507855638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507855638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43528 deadline: 1679507855641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,666 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:35,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:35,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:35,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:35,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:35,669 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:35,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:35,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:35,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,670 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,671 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:35,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:35,672 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:35,672 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,672 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:35,672 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:35,672 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:35,673 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,673 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:35,673 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:35,673 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:35,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:35,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:35,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:35,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,677 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,677 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:35,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ce8c212[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:35,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:35,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,681 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:35,682 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:35,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:35,685 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:35,685 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:35,685 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:35,685 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:35,685 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:35,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,686 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,689 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,692 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:35,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:35,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,692 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:35,692 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:35,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:35,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/edeb0af0b16642a59d864fcb6b9d1eec 2023-03-22 17:56:35,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/252c56e0823445a2b0904efebebf714b 2023-03-22 17:56:35,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/c1bf165b419943ecbca9be3b51bd7b7a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c1bf165b419943ecbca9be3b51bd7b7a 2023-03-22 17:56:35,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c1bf165b419943ecbca9be3b51bd7b7a, entries=200, sequenceid=518, filesize=32.3 K 2023-03-22 17:56:35,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/edeb0af0b16642a59d864fcb6b9d1eec as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/edeb0af0b16642a59d864fcb6b9d1eec 2023-03-22 17:56:35,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/edeb0af0b16642a59d864fcb6b9d1eec, entries=150, sequenceid=518, filesize=11.9 K 2023-03-22 17:56:35,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/252c56e0823445a2b0904efebebf714b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/252c56e0823445a2b0904efebebf714b 2023-03-22 17:56:35,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/252c56e0823445a2b0904efebebf714b, entries=150, sequenceid=518, filesize=11.9 K 2023-03-22 17:56:35,766 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for 876c388f3b400624afff0881af885f88 in 1259ms, sequenceid=518, compaction requested=false 2023-03-22 17:56:35,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:35,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:35,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:35,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:35,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:35,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:35,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:35,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:35,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:35,792 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:35,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:35,794 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322e4bbbad0804549cc9773a4a645c90558 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e4bbbad0804549cc9773a4a645c90558 2023-03-22 17:56:35,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:35,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:35,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:35,796 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:35,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:35,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:35,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,799 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,799 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,800 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,800 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:35,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:35,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:35,804 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=531, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/1ac72d1072ce4057a807fd36c31c6306 2023-03-22 17:56:35,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:35,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:35,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:35,804 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:35,804 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,805 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:35,805 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:35,805 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:35,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:35,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:35,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:35,807 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,807 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,807 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,807 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:35,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@352978c5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:35,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:35,810 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507795797, max timestamp=1679507795808Num uniques 859; 2023-03-22 17:56:35,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:35,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507795797, max timestamp=1679507795808Num uniques 859; 2023-03-22 17:56:35,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:35,812 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:35,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:35,812 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,812 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:35,812 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:35,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507795797, max timestamp=1679507795808Num uniques 859; 2023-03-22 17:56:35,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:35,814 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:35,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/81893cc5ead84071b7c3e909965c7c0e 2023-03-22 17:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:35,818 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:35,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:35,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:35,818 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:35,818 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:35,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:35,821 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:35,821 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,823 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,826 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,826 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,826 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,826 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:35,827 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:35,826 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:35,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:35,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:35,827 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,827 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:35,827 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:35,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:35,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:35,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/cbf8c80a14354a58b9556d3e53a8e13a 2023-03-22 17:56:35,840 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/1ac72d1072ce4057a807fd36c31c6306 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1ac72d1072ce4057a807fd36c31c6306 2023-03-22 17:56:35,843 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1ac72d1072ce4057a807fd36c31c6306, entries=150, sequenceid=531, filesize=25.4 K 2023-03-22 17:56:35,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/81893cc5ead84071b7c3e909965c7c0e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/81893cc5ead84071b7c3e909965c7c0e 2023-03-22 17:56:35,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/81893cc5ead84071b7c3e909965c7c0e, entries=150, sequenceid=531, filesize=11.9 K 2023-03-22 17:56:35,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507855842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507855844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/cbf8c80a14354a58b9556d3e53a8e13a as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/cbf8c80a14354a58b9556d3e53a8e13a 2023-03-22 17:56:35,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/cbf8c80a14354a58b9556d3e53a8e13a, entries=150, sequenceid=531, filesize=11.9 K 2023-03-22 17:56:35,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for 876c388f3b400624afff0881af885f88 in 72ms, sequenceid=531, compaction requested=true 2023-03-22 17:56:35,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:35,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:35,852 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:35,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:35,852 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:35,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:35,852 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:35,852 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:35,853 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:35,853 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,853 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:35,853 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4b0f2adbc4364909aee8f408cd648fa9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c1bf165b419943ecbca9be3b51bd7b7a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1ac72d1072ce4057a807fd36c31c6306] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=84.1 K 2023-03-22 17:56:35,853 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,853 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b85c4cf60982458f8c742411e651b7de, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/edeb0af0b16642a59d864fcb6b9d1eec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/81893cc5ead84071b7c3e909965c7c0e] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=36.6 K 2023-03-22 17:56:35,853 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 4b0f2adbc4364909aee8f408cd648fa9, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1679507792900 2023-03-22 17:56:35,853 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting b85c4cf60982458f8c742411e651b7de, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1679507792900 2023-03-22 17:56:35,853 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting c1bf165b419943ecbca9be3b51bd7b7a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1679507793640 2023-03-22 17:56:35,854 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting edeb0af0b16642a59d864fcb6b9d1eec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1679507793640 2023-03-22 17:56:35,854 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 1ac72d1072ce4057a807fd36c31c6306, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1679507794511 2023-03-22 17:56:35,854 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 81893cc5ead84071b7c3e909965c7c0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1679507794511 2023-03-22 17:56:35,860 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#622 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:35,865 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#621 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:35,882 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/d0191d66c60843b08b61b77cce5468ed as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d0191d66c60843b08b61b77cce5468ed 2023-03-22 17:56:35,888 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into d0191d66c60843b08b61b77cce5468ed(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:35,888 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:35,888 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=13, startTime=1679507795852; duration=0sec 2023-03-22 17:56:35,888 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:35,888 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:35,890 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:35,890 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:35,890 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,890 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2dc288712eea48c08af07e1d1f94403c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/252c56e0823445a2b0904efebebf714b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/cbf8c80a14354a58b9556d3e53a8e13a] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=36.6 K 2023-03-22 17:56:35,890 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2dc288712eea48c08af07e1d1f94403c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1679507792900 2023-03-22 17:56:35,890 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 252c56e0823445a2b0904efebebf714b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1679507793640 2023-03-22 17:56:35,891 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting cbf8c80a14354a58b9556d3e53a8e13a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1679507794511 2023-03-22 17:56:35,895 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/a98035f7f867435cb886505fc2983ca1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/a98035f7f867435cb886505fc2983ca1 2023-03-22 17:56:35,895 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#623 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:35,898 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into a98035f7f867435cb886505fc2983ca1(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:35,898 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:35,898 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=13, startTime=1679507795851; duration=0sec 2023-03-22 17:56:35,899 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:35,927 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:35,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:35,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:35,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:35,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:35,930 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:35,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:35,930 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:35,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,932 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,932 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:35,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:35,933 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,933 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:35,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:35,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:35,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:35,934 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:35,934 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:35,934 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:35,935 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:35,935 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:35,935 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:35,935 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:35,935 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-03-22 17:56:35,935 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:35,935 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:35,936 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:35,936 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:35,936 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:35,936 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:35,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:35,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:35,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507855963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:35,966 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322ca86ffda1e994be18bfac3ea6dd67cd8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ca86ffda1e994be18bfac3ea6dd67cd8 2023-03-22 17:56:35,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:35,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507855965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507856066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507856069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507856270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507856271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,309 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/6f41224f5ec94db184bf33c7481da4f9 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6f41224f5ec94db184bf33c7481da4f9 2023-03-22 17:56:36,313 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into 6f41224f5ec94db184bf33c7481da4f9(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:36,313 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:36,313 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=13, startTime=1679507795852; duration=0sec 2023-03-22 17:56:36,313 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:36,377 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=569, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/55cbd74687694fed938a9d786cfa195e 2023-03-22 17:56:36,385 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/af9d070eedfa4b1886e5dfefb4566e4f 2023-03-22 17:56:36,397 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/6cc7bdebd6f94d5f88970652fe1c5585 2023-03-22 17:56:36,400 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/55cbd74687694fed938a9d786cfa195e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/55cbd74687694fed938a9d786cfa195e 2023-03-22 17:56:36,403 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/55cbd74687694fed938a9d786cfa195e, entries=150, sequenceid=569, filesize=25.4 K 2023-03-22 17:56:36,404 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/af9d070eedfa4b1886e5dfefb4566e4f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/af9d070eedfa4b1886e5dfefb4566e4f 2023-03-22 17:56:36,408 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/af9d070eedfa4b1886e5dfefb4566e4f, entries=150, sequenceid=569, filesize=11.9 K 2023-03-22 17:56:36,408 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/6cc7bdebd6f94d5f88970652fe1c5585 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6cc7bdebd6f94d5f88970652fe1c5585 2023-03-22 17:56:36,411 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6cc7bdebd6f94d5f88970652fe1c5585, entries=150, sequenceid=569, filesize=11.9 K 2023-03-22 17:56:36,412 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for 876c388f3b400624afff0881af885f88 in 477ms, sequenceid=569, compaction requested=false 2023-03-22 17:56:36,412 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:36,412 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:36,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:36,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:36,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:36,412 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:36,413 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:36,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:36,414 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:36,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:36,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f889328[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:36,416 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:36,416 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:36,419 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:36,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:36,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:36,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:36,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:36,420 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,420 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:36,420 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:36,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:36,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:36,421 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,421 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:36,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:36,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:36,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:36,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:36,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:36,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:36,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:36,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:36,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:36,424 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:36,424 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:36,424 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:36,424 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:36,424 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:36,426 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:36,426 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,426 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:36,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,426 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:36,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:36,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:36,428 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:36,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:36,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,430 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:36,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:36,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:36,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:36,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:36,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:36,430 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:36,431 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:36,431 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,431 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:36,431 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:36,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:36,431 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:36,431 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:36,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:36,431 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:36,531 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:36,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:36,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:36,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:36,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:36,534 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:36,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:36,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:36,534 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:36,535 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,535 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:36,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,537 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:36,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:36,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,537 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:36,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:36,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:36,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:36,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:36,538 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:36,538 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:36,538 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:36,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:36,539 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:36,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:36,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:36,539 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:36,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:36,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:36,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:36,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:36,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:36,539 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:36,560 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322d2e9bf6567664954943071033fcc0352 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322d2e9bf6567664954943071033fcc0352 2023-03-22 17:56:36,574 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=580, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/e62d89936878417289903a6738768af1 2023-03-22 17:56:36,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:36,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:36,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507796589, max timestamp=1679507796604Num uniques 859; 2023-03-22 17:56:36,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:36,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679507796589, max timestamp=1679507796604Num uniques 859; 2023-03-22 17:56:36,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-22 17:56:36,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507796589, max timestamp=1679507796604Num uniques 859; 2023-03-22 17:56:36,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:36,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507856648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43456 deadline: 1679507856651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507856653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507856653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507856654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507856754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507856756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507856758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507856758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507856957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507856960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507856962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:36,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507856964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:36,983 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/2528297abf074903a6f78860b69faff3 2023-03-22 17:56:36,992 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/f270309e123142a49ea04f935ce0f8fc 2023-03-22 17:56:36,995 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/e62d89936878417289903a6738768af1 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e62d89936878417289903a6738768af1 2023-03-22 17:56:36,998 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e62d89936878417289903a6738768af1, entries=150, sequenceid=580, filesize=25.4 K 2023-03-22 17:56:36,998 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/2528297abf074903a6f78860b69faff3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2528297abf074903a6f78860b69faff3 2023-03-22 17:56:37,001 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2528297abf074903a6f78860b69faff3, entries=150, sequenceid=580, filesize=11.9 K 2023-03-22 17:56:37,002 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/f270309e123142a49ea04f935ce0f8fc as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f270309e123142a49ea04f935ce0f8fc 2023-03-22 17:56:37,005 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f270309e123142a49ea04f935ce0f8fc, entries=150, sequenceid=580, filesize=11.9 K 2023-03-22 17:56:37,005 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.65 KB/141980 for 876c388f3b400624afff0881af885f88 in 466ms, sequenceid=580, compaction requested=true 2023-03-22 17:56:37,006 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:37,006 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:37,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:37,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:37,006 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:37,007 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,007 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,008 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,008 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:37,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26784549[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:37,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:37,010 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,011 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:37,011 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,011 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:37,016 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:37,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,016 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:37,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:37,018 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:37,018 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:37,018 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:37,018 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:37,018 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:37,020 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,020 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,020 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,020 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,021 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:37,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:37,024 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,024 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,024 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:37,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,024 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:37,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,124 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:37,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:37,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:37,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:37,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:37,128 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:37,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:37,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:37,128 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,129 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,130 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,130 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,131 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:37,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:37,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:37,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:37,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:37,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:37,132 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:37,132 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,132 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:37,132 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-03-22 17:56:37,133 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:37,133 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:37,133 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:37,133 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:37,133 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:37,133 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:37,145 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032217a43faa6669439381d7d39675c22c66 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032217a43faa6669439381d7d39675c22c66 2023-03-22 17:56:37,156 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=617, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/687b3ba35f894473b36490ec06c2f75f 2023-03-22 17:56:37,166 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/7400c295c76e44b7840e9009c7a7ba48 2023-03-22 17:56:37,180 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/7a4df21464f048108c579bf022c8b147 2023-03-22 17:56:37,184 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/687b3ba35f894473b36490ec06c2f75f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/687b3ba35f894473b36490ec06c2f75f 2023-03-22 17:56:37,189 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/687b3ba35f894473b36490ec06c2f75f, entries=150, sequenceid=617, filesize=25.4 K 2023-03-22 17:56:37,189 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/7400c295c76e44b7840e9009c7a7ba48 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/7400c295c76e44b7840e9009c7a7ba48 2023-03-22 17:56:37,193 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/7400c295c76e44b7840e9009c7a7ba48, entries=150, sequenceid=617, filesize=11.9 K 2023-03-22 17:56:37,194 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/7a4df21464f048108c579bf022c8b147 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7a4df21464f048108c579bf022c8b147 2023-03-22 17:56:37,205 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7a4df21464f048108c579bf022c8b147, entries=150, sequenceid=617, filesize=11.9 K 2023-03-22 17:56:37,206 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=0 B/0 for 876c388f3b400624afff0881af885f88 in 74ms, sequenceid=617, compaction requested=true 2023-03-22 17:56:37,206 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:37,206 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,206 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:37,206 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:37,206 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,206 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:37,206 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:37,208 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,208 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,209 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,209 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:37,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8afbc6[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:37,211 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:37,211 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,212 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:37,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,212 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:37,213 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:37,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:37,216 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:37,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:37,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:37,216 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:37,216 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:37,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,218 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,218 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,220 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,226 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:37,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:37,226 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:37,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,227 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:37,227 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,227 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:37,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:37,297 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:37,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:37,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:37,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:37,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:37,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:37,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:37,327 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:37,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:37,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:37,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507797323, max timestamp=1679507797330Num uniques 859; 2023-03-22 17:56:37,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:37,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507797323, max timestamp=1679507797329Num uniques 859; 2023-03-22 17:56:37,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:37,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679507797323, max timestamp=1679507797330Num uniques 859; 2023-03-22 17:56:37,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-22 17:56:37,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:37,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:37,333 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:37,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:37,333 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:37,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,334 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,336 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,336 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,336 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,336 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:37,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:37,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:37,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:37,338 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,338 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:37,338 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:37,338 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,338 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322ce3683ec70c64c28bae1d4954e647181 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ce3683ec70c64c28bae1d4954e647181 2023-03-22 17:56:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:37,339 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:37,341 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,341 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,341 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,341 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:37,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@521b51c2[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:37,343 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:37,343 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,344 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:37,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,345 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:37,347 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,347 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=628, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/b1134614e15145f8980a8b0e86d2a235 2023-03-22 17:56:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:37,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:37,351 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:37,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:37,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:37,351 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:37,351 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:37,353 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,353 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,353 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,355 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,357 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,359 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:37,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:37,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:37,359 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,360 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,360 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:37,360 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:37,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/21c3647e8e3e4a168465072199a33e54 2023-03-22 17:56:37,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507857368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507857369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507857369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507857370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/14341fefd9a2447eadb3d49983e31430 2023-03-22 17:56:37,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/b1134614e15145f8980a8b0e86d2a235 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b1134614e15145f8980a8b0e86d2a235 2023-03-22 17:56:37,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b1134614e15145f8980a8b0e86d2a235, entries=300, sequenceid=628, filesize=46.1 K 2023-03-22 17:56:37,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/21c3647e8e3e4a168465072199a33e54 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/21c3647e8e3e4a168465072199a33e54 2023-03-22 17:56:37,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/21c3647e8e3e4a168465072199a33e54, entries=150, sequenceid=628, filesize=11.9 K 2023-03-22 17:56:37,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/14341fefd9a2447eadb3d49983e31430 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/14341fefd9a2447eadb3d49983e31430 2023-03-22 17:56:37,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/14341fefd9a2447eadb3d49983e31430, entries=150, sequenceid=628, filesize=11.9 K 2023-03-22 17:56:37,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 876c388f3b400624afff0881af885f88 in 94ms, sequenceid=628, compaction requested=true 2023-03-22 17:56:37,391 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:37,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:37,391 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:37,391 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:37,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:37,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:37,393 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 152376 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:37,393 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:37,393 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,393 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/a98035f7f867435cb886505fc2983ca1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/55cbd74687694fed938a9d786cfa195e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e62d89936878417289903a6738768af1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/687b3ba35f894473b36490ec06c2f75f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b1134614e15145f8980a8b0e86d2a235] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=148.8 K 2023-03-22 17:56:37,393 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61952 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:37,393 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting a98035f7f867435cb886505fc2983ca1, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1679507794511 2023-03-22 17:56:37,393 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:37,393 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,393 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d0191d66c60843b08b61b77cce5468ed, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/af9d070eedfa4b1886e5dfefb4566e4f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2528297abf074903a6f78860b69faff3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/7400c295c76e44b7840e9009c7a7ba48, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/21c3647e8e3e4a168465072199a33e54] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=60.5 K 2023-03-22 17:56:37,394 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 55cbd74687694fed938a9d786cfa195e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679507795836 2023-03-22 17:56:37,394 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d0191d66c60843b08b61b77cce5468ed, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1679507794511 2023-03-22 17:56:37,394 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting e62d89936878417289903a6738768af1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1679507795955 2023-03-22 17:56:37,394 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting af9d070eedfa4b1886e5dfefb4566e4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679507795836 2023-03-22 17:56:37,394 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 687b3ba35f894473b36490ec06c2f75f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1679507796624 2023-03-22 17:56:37,394 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2528297abf074903a6f78860b69faff3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1679507795955 2023-03-22 17:56:37,395 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b1134614e15145f8980a8b0e86d2a235, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1679507797276 2023-03-22 17:56:37,395 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7400c295c76e44b7840e9009c7a7ba48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1679507796624 2023-03-22 17:56:37,395 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 21c3647e8e3e4a168465072199a33e54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1679507797294 2023-03-22 17:56:37,417 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#637 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:37,419 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#636 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:37,429 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/86cb9406ef934e5a88d0777c0f118462 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/86cb9406ef934e5a88d0777c0f118462 2023-03-22 17:56:37,433 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into 86cb9406ef934e5a88d0777c0f118462(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:37,433 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:37,433 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=11, startTime=1679507797391; duration=0sec 2023-03-22 17:56:37,433 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:37,433 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-22 17:56:37,434 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61952 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-22 17:56:37,435 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:37,435 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,435 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6f41224f5ec94db184bf33c7481da4f9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6cc7bdebd6f94d5f88970652fe1c5585, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f270309e123142a49ea04f935ce0f8fc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7a4df21464f048108c579bf022c8b147, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/14341fefd9a2447eadb3d49983e31430] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=60.5 K 2023-03-22 17:56:37,435 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6f41224f5ec94db184bf33c7481da4f9, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1679507794511 2023-03-22 17:56:37,435 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 6cc7bdebd6f94d5f88970652fe1c5585, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1679507795836 2023-03-22 17:56:37,436 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting f270309e123142a49ea04f935ce0f8fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1679507795955 2023-03-22 17:56:37,436 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 7a4df21464f048108c579bf022c8b147, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1679507796624 2023-03-22 17:56:37,436 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 14341fefd9a2447eadb3d49983e31430, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1679507797294 2023-03-22 17:56:37,444 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#638 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:37,460 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:37,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:37,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:37,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:37,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:37,463 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:37,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:37,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:37,463 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,464 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,465 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,465 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:37,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:37,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:37,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:37,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,468 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:37,472 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:37,472 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:37,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:37,472 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-22 17:56:37,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:37,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:37,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:37,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:37,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:37,472 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-22 17:56:37,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:37,480 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:37,486 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322578016ee11734746aa962cf72b5550b7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322578016ee11734746aa962cf72b5550b7 2023-03-22 17:56:37,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507857485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507857486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507857486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507857487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,501 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=665, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/5e3f4c5c9174400fae88d484e01f79c8 2023-03-22 17:56:37,513 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/3d26d023a6d5449e9108741f0eb910c0 2023-03-22 17:56:37,524 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/2da092be0b584a54924651988e5f3837 2023-03-22 17:56:37,528 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/5e3f4c5c9174400fae88d484e01f79c8 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/5e3f4c5c9174400fae88d484e01f79c8 2023-03-22 17:56:37,531 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/5e3f4c5c9174400fae88d484e01f79c8, entries=150, sequenceid=665, filesize=25.4 K 2023-03-22 17:56:37,532 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/3d26d023a6d5449e9108741f0eb910c0 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3d26d023a6d5449e9108741f0eb910c0 2023-03-22 17:56:37,536 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3d26d023a6d5449e9108741f0eb910c0, entries=150, sequenceid=665, filesize=11.9 K 2023-03-22 17:56:37,537 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/2da092be0b584a54924651988e5f3837 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2da092be0b584a54924651988e5f3837 2023-03-22 17:56:37,539 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2da092be0b584a54924651988e5f3837, entries=150, sequenceid=665, filesize=11.9 K 2023-03-22 17:56:37,540 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 876c388f3b400624afff0881af885f88 in 68ms, sequenceid=665, compaction requested=false 2023-03-22 17:56:37,540 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:37,540 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,540 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:37,540 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:37,540 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,540 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:37,540 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:37,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,543 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,543 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:37,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39e1897a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:37,545 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:37,545 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,546 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:37,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:37,546 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:37,547 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,548 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:37,549 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,549 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:37,549 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,549 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:37,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:37,551 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:37,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:37,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:37,552 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:37,552 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:37,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,557 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,558 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,563 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,563 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:37,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:37,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:37,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,564 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:37,564 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:37,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:37,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-22 17:56:37,594 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:37,594 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:37,594 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:37,594 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:37,594 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:37,594 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:37,604 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032238b7e901ddc24718b380374285e72b47 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032238b7e901ddc24718b380374285e72b47 2023-03-22 17:56:37,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507857611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507857612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507857613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507857615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,664 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:37,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:37,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:37,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:37,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:37,668 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:37,668 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,668 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:37,668 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:37,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,669 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,671 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,671 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:37,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:37,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:37,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:37,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:37,673 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:37,673 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,673 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:37,673 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:37,674 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,675 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,676 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:37,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dd7d102[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:37,678 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:37,678 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,679 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,679 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:37,681 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,681 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:37,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:37,684 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:37,684 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:37,684 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:37,684 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:37,684 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:37,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,687 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,688 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,689 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,691 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,692 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:37,692 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:37,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:37,693 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:37,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,693 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:37,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507857717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507857717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507857717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507857720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,793 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:37,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:37,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:37,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:37,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:37,796 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:37,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:37,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:37,796 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,797 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,798 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,798 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:37,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:37,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:37,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:37,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:37,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:37,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:37,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:37,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:37,799 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,799 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:37,799 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:37,799 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:37,799 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:37,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:37,800 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:37,801 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,801 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,801 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,801 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267845b5[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:37,802 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:37,803 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,805 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,805 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:37,805 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:37,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:37,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,806 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:37,806 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:37,809 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:37,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:37,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:37,809 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:37,809 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:37,810 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,810 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,811 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,811 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,812 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,814 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:37,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:37,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:37,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,815 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,815 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:37,815 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:37,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,815 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,853 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d164a9b1a11244009bb508f8981a1feb as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d164a9b1a11244009bb508f8981a1feb 2023-03-22 17:56:37,856 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into d164a9b1a11244009bb508f8981a1feb(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:37,857 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:37,857 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=11, startTime=1679507797391; duration=0sec 2023-03-22 17:56:37,857 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:37,915 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:37,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:37,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:37,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:37,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:37,918 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:37,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:37,918 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:37,919 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,919 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,921 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,921 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507857919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,921 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,921 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:37,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507857919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507857921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:37,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:37,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:37,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:37,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:37,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:37,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:37,923 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:37,923 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,923 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:37,923 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:37,924 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:37,924 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:37,924 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:37,924 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,924 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:37,924 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,926 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:37,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507857924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,926 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ee585f9[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:37,929 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:37,929 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,930 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:37,931 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:37,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,931 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:37,931 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:37,932 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:37,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,932 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:37,935 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:37,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:37,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:37,935 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:37,935 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:37,937 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,937 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,937 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:37,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,938 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:37,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,939 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:37,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:37,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:37,943 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:37,943 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:37,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:37,943 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:37,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:37,944 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:37,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,009 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=677, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/eff80674a15a4ab28ec95a2f218dbfda 2023-03-22 17:56:38,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/bd88762651d24c90ad2d1bca33349137 2023-03-22 17:56:38,044 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:38,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:38,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:38,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:38,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:38,047 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:38,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:38,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:38,047 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,048 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,049 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,049 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:38,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:38,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:38,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:38,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:38,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,050 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:38,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:38,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:38,051 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,051 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:38,051 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:38,051 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:38,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:38,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:38,051 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:38,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b7a6306[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:38,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:38,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,057 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,057 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:38,057 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:38,059 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:38,059 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:38,061 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:38,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:38,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:38,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:38,061 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:38,063 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,063 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,063 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:38,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:38,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:38,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:38,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,069 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:38,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,069 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:38,069 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:38,169 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:38,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:38,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:38,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:38,173 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:38,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:38,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:38,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,173 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,175 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,175 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,176 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,176 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:38,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:38,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:38,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:38,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:38,177 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:38,177 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,177 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:38,177 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:38,177 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:38,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,179 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:38,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c6eeaa[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:38,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:38,182 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,183 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:38,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,183 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:38,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:38,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:38,185 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,185 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:38,188 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:38,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:38,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:38,188 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:38,188 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:38,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,194 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,195 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,197 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,199 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:38,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:38,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:38,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:38,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:38,199 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,200 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:38,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,200 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:38,200 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:38,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507858222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507858223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507858225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507858227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,250 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/0c238f866f3442a1be87429d345f0292 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/0c238f866f3442a1be87429d345f0292 2023-03-22 17:56:38,255 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into 0c238f866f3442a1be87429d345f0292(size=26.6 K), total size for store is 52.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:38,255 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:38,255 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=11, startTime=1679507797391; duration=0sec 2023-03-22 17:56:38,255 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:38,300 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:38,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:38,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:38,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:38,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:38,305 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:38,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:38,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:38,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,305 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,307 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,307 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:38,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:38,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:38,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:38,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:38,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,308 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:38,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:38,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:38,309 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,309 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:38,309 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:38,309 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:38,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:38,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:38,309 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:38,311 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,311 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf759ea[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:38,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:38,313 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,314 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:38,314 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:38,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,315 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:38,317 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:38,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,317 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:38,320 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:38,320 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:38,320 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:38,321 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:38,321 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:38,322 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,322 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,322 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,323 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,324 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,327 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:38,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:38,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,327 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,327 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,327 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,427 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:38,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:38,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:38,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:38,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:38,431 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:38,431 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,431 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:38,431 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:38,431 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,431 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,435 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/1f249f90d76d4e69af20853ea1609136 2023-03-22 17:56:38,436 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,436 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,436 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,436 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:38,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:38,437 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:38,437 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:38,438 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,438 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:38,438 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:38,438 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:38,438 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:38,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/eff80674a15a4ab28ec95a2f218dbfda as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/eff80674a15a4ab28ec95a2f218dbfda 2023-03-22 17:56:38,440 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,440 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,440 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,440 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:38,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/eff80674a15a4ab28ec95a2f218dbfda, entries=200, sequenceid=677, filesize=32.3 K 2023-03-22 17:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ebfddc0[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:38,443 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:38,443 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/bd88762651d24c90ad2d1bca33349137 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bd88762651d24c90ad2d1bca33349137 2023-03-22 17:56:38,444 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:38,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:38,444 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:38,444 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,445 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:38,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bd88762651d24c90ad2d1bca33349137, entries=150, sequenceid=677, filesize=11.9 K 2023-03-22 17:56:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:38,447 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/1f249f90d76d4e69af20853ea1609136 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/1f249f90d76d4e69af20853ea1609136 2023-03-22 17:56:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:38,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:38,450 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:38,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:38,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:38,451 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:38,451 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:38,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/1f249f90d76d4e69af20853ea1609136, entries=150, sequenceid=677, filesize=11.9 K 2023-03-22 17:56:38,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 876c388f3b400624afff0881af885f88 in 859ms, sequenceid=677, compaction requested=true 2023-03-22 17:56:38,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:38,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:38,452 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:38,452 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:38,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:38,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:38,453 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,453 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:38,453 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:38,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,454 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86370 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:38,454 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:38,454 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:38,454 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/0c238f866f3442a1be87429d345f0292, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/5e3f4c5c9174400fae88d484e01f79c8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/eff80674a15a4ab28ec95a2f218dbfda] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=84.3 K 2023-03-22 17:56:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,454 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:38,454 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:38,454 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,454 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 0c238f866f3442a1be87429d345f0292, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1679507797294 2023-03-22 17:56:38,454 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/86cb9406ef934e5a88d0777c0f118462, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3d26d023a6d5449e9108741f0eb910c0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bd88762651d24c90ad2d1bca33349137] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=36.9 K 2023-03-22 17:56:38,455 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:38,455 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 5e3f4c5c9174400fae88d484e01f79c8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1679507797356 2023-03-22 17:56:38,455 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 86cb9406ef934e5a88d0777c0f118462, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1679507797294 2023-03-22 17:56:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,455 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting eff80674a15a4ab28ec95a2f218dbfda, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679507797479 2023-03-22 17:56:38,455 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 3d26d023a6d5449e9108741f0eb910c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1679507797356 2023-03-22 17:56:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:38,456 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting bd88762651d24c90ad2d1bca33349137, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679507797479 2023-03-22 17:56:38,457 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,461 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,461 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:38,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:38,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,462 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:38,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:38,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:38,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:38,462 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,462 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:38,462 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:38,465 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#646 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:38,469 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#645 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:38,492 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/591831f5e54c4771b6f0604471ea520c as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/591831f5e54c4771b6f0604471ea520c 2023-03-22 17:56:38,496 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into 591831f5e54c4771b6f0604471ea520c(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:38,496 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:38,496 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=13, startTime=1679507798452; duration=0sec 2023-03-22 17:56:38,496 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:38,496 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-22 17:56:38,497 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-22 17:56:38,497 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:38,497 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,497 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d164a9b1a11244009bb508f8981a1feb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2da092be0b584a54924651988e5f3837, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/1f249f90d76d4e69af20853ea1609136] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=36.9 K 2023-03-22 17:56:38,498 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d164a9b1a11244009bb508f8981a1feb, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1679507797294 2023-03-22 17:56:38,498 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 2da092be0b584a54924651988e5f3837, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1679507797356 2023-03-22 17:56:38,498 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1f249f90d76d4e69af20853ea1609136, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679507797479 2023-03-22 17:56:38,505 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#647 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:38,518 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/54981c27e5854c159c8b7872ff941fca as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/54981c27e5854c159c8b7872ff941fca 2023-03-22 17:56:38,522 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into 54981c27e5854c159c8b7872ff941fca(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:38,522 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:38,522 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=13, startTime=1679507798452; duration=0sec 2023-03-22 17:56:38,522 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:38,562 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:38,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:38,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:38,566 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:38,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:38,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:38,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,566 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,568 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:38,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:38,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,568 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:38,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:38,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:38,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:38,569 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:38,569 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:38,569 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:38,570 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:38,570 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:38,570 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:38,570 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:38,570 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-22 17:56:38,570 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:38,570 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:38,570 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:38,570 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:38,570 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:38,570 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:38,593 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322ed4ab677b75b42fc94b7726d1999435a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ed4ab677b75b42fc94b7726d1999435a 2023-03-22 17:56:38,599 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=701, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/91ef6c7fca8f47aa9cc0af5d7379084b 2023-03-22 17:56:38,607 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/eb3acfc8bb154bb68f85efb77397245e 2023-03-22 17:56:38,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:38,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:38,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43456 deadline: 1679507858685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43528 deadline: 1679507858727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43504 deadline: 1679507858728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43486 deadline: 1679507858729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34403] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1679507858730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43456 deadline: 1679507858789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:38,909 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/4042ab1dd36f4a088de734f470c1ee34 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4042ab1dd36f4a088de734f470c1ee34 2023-03-22 17:56:38,913 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into 4042ab1dd36f4a088de734f470c1ee34(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:38,913 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:38,913 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=13, startTime=1679507798452; duration=0sec 2023-03-22 17:56:38,913 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:38,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-22 17:56:38,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43456 deadline: 1679507858992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,018 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/06b60623c7c6415fa0bd2103a438c78e 2023-03-22 17:56:39,031 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/91ef6c7fca8f47aa9cc0af5d7379084b as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91ef6c7fca8f47aa9cc0af5d7379084b 2023-03-22 17:56:39,034 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91ef6c7fca8f47aa9cc0af5d7379084b, entries=150, sequenceid=701, filesize=25.4 K 2023-03-22 17:56:39,035 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/eb3acfc8bb154bb68f85efb77397245e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/eb3acfc8bb154bb68f85efb77397245e 2023-03-22 17:56:39,038 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/eb3acfc8bb154bb68f85efb77397245e, entries=150, sequenceid=701, filesize=11.9 K 2023-03-22 17:56:39,039 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/06b60623c7c6415fa0bd2103a438c78e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/06b60623c7c6415fa0bd2103a438c78e 2023-03-22 17:56:39,048 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/06b60623c7c6415fa0bd2103a438c78e, entries=150, sequenceid=701, filesize=11.9 K 2023-03-22 17:56:39,049 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 876c388f3b400624afff0881af885f88 in 479ms, sequenceid=701, compaction requested=false 2023-03-22 17:56:39,049 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:39,049 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:39,052 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,052 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,053 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:39,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5695ff42[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:39,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:39,054 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,055 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:39,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,056 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:39,057 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:39,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:39,061 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:39,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:39,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:39,061 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:39,061 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:39,063 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,063 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,063 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,064 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,065 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,067 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:39,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:39,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:39,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:39,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:39,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:39,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,068 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:39,068 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,068 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:39,068 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:39,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:39,168 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:39,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:39,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:39,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:39,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:39,178 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:39,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:39,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:39,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:39,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,178 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,179 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:39,179 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:39,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:39,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,180 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:39,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:39,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:39,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:39,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:39,181 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:39,181 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:39,181 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-22 17:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:39,182 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,196 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322828458341b5d44d1a108a079d05b199d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322828458341b5d44d1a108a079d05b199d 2023-03-22 17:56:39,201 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=714, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/b470c909202042b98607068221bd6ef5 2023-03-22 17:56:39,209 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/1199c3adb24543de9b48b8c0c15516a3 2023-03-22 17:56:39,219 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d2592b58f9e748b9ac8576b36fd0ae44 2023-03-22 17:56:39,224 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/b470c909202042b98607068221bd6ef5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b470c909202042b98607068221bd6ef5 2023-03-22 17:56:39,228 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b470c909202042b98607068221bd6ef5, entries=100, sequenceid=714, filesize=18.6 K 2023-03-22 17:56:39,228 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/1199c3adb24543de9b48b8c0c15516a3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/1199c3adb24543de9b48b8c0c15516a3 2023-03-22 17:56:39,232 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/1199c3adb24543de9b48b8c0c15516a3, entries=100, sequenceid=714, filesize=9.5 K 2023-03-22 17:56:39,232 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/d2592b58f9e748b9ac8576b36fd0ae44 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d2592b58f9e748b9ac8576b36fd0ae44 2023-03-22 17:56:39,236 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d2592b58f9e748b9ac8576b36fd0ae44, entries=100, sequenceid=714, filesize=9.5 K 2023-03-22 17:56:39,237 INFO [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 876c388f3b400624afff0881af885f88 in 56ms, sequenceid=714, compaction requested=true 2023-03-22 17:56:39,237 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:39,237 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:39,238 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,238 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,239 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1969e884[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:39,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:39,241 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,242 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:39,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,242 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:39,247 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:39,247 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:39,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:39,249 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:39,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:39,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:39,249 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:39,249 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:39,251 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,252 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,254 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:39,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,255 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:39,258 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:39,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:39,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:39,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,258 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,258 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:39,258 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:39,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:39,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:39,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-22 17:56:39,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:39,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:39,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:39,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,358 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:39,359 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032209085a59b4b74f48a14755905ccd4a8d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032209085a59b4b74f48a14755905ccd4a8d 2023-03-22 17:56:39,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:39,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-22 17:56:39,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-22 17:56:39,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-22 17:56:39,365 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-22 17:56:39,365 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:39,365 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-22 17:56:39,365 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-22 17:56:39,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,366 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,367 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:39,367 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:39,367 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=724, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/32e58fb00190484d9801efa2139a3e27 2023-03-22 17:56:39,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,368 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-22 17:56:39,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-22 17:56:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-22 17:56:39,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-22 17:56:39,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-22 17:56:39,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-22 17:56:39,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-22 17:56:39,369 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-22 17:56:39,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-22 17:56:39,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-22 17:56:39,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:39,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. started... 2023-03-22 17:56:39,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. as already flushing 2023-03-22 17:56:39,370 DEBUG [rs(jenkins-hbase4.apache.org,34403,1679507617723)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:39,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-22 17:56:39,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-22 17:56:39,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-22 17:56:39,370 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-22 17:56:39,373 DEBUG [Thread-4024] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eedd898 to 127.0.0.1:55285 2023-03-22 17:56:39,373 DEBUG [Thread-4024] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,374 DEBUG [Thread-4016] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:55285 2023-03-22 17:56:39,374 DEBUG [Thread-4016] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,375 DEBUG [Thread-4022] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:55285 2023-03-22 17:56:39,375 DEBUG [Thread-4022] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,375 DEBUG [Thread-4007] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:55285 2023-03-22 17:56:39,376 DEBUG [Thread-4007] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,376 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/9cbde7f301c946209466cbd47c4af4ab 2023-03-22 17:56:39,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,377 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-22 17:56:39,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,378 DEBUG [Thread-4018] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:55285 2023-03-22 17:56:39,378 DEBUG [Thread-4018] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,379 DEBUG [Thread-4020] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:55285 2023-03-22 17:56:39,379 DEBUG [Thread-4020] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34403,1679507617723' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-22 17:56:39,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff7ca4a[Count = 0] remaining members to acquire global barrier 2023-03-22 17:56:39,383 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-22 17:56:39,383 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,384 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-22 17:56:39,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,384 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-22 17:56:39,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-22 17:56:39,384 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34403,1679507617723' in zk 2023-03-22 17:56:39,386 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-22 17:56:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,34403,1679507617723' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-22 17:56:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34403,1679507617723': 2023-03-22 17:56:39,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34403,1679507617723' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-22 17:56:39,388 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-22 17:56:39,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-22 17:56:39,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-22 17:56:39,388 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-22 17:56:39,388 INFO [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-22 17:56:39,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-22 17:56:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-22 17:56:39,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,391 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,391 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-22 17:56:39,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-22 17:56:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,393 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-22 17:56:39,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=724 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/985d838f9c214df8a90ec8ca088e20c2 2023-03-22 17:56:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-22 17:56:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-22 17:56:39,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-22 17:56:39,395 DEBUG [(jenkins-hbase4.apache.org,40203,1679507616414)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-22 17:56:39,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-22 17:56:39,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-22 17:56:39,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-22 17:56:39,395 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-22 17:56:39,396 DEBUG [Thread-4015] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-22 17:56:39,396 DEBUG [Thread-4015] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-22 17:56:39,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/32e58fb00190484d9801efa2139a3e27 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/32e58fb00190484d9801efa2139a3e27 2023-03-22 17:56:39,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-22 17:56:39,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/32e58fb00190484d9801efa2139a3e27, entries=200, sequenceid=724, filesize=32.3 K 2023-03-22 17:56:39,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-22 17:56:39,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-22 17:56:39,400 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-22 17:56:39,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-22 17:56:39,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-22 17:56:39,400 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/9cbde7f301c946209466cbd47c4af4ab as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9cbde7f301c946209466cbd47c4af4ab 2023-03-22 17:56:39,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9cbde7f301c946209466cbd47c4af4ab, entries=150, sequenceid=724, filesize=11.9 K 2023-03-22 17:56:39,404 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/985d838f9c214df8a90ec8ca088e20c2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/985d838f9c214df8a90ec8ca088e20c2 2023-03-22 17:56:39,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/985d838f9c214df8a90ec8ca088e20c2, entries=150, sequenceid=724, filesize=11.9 K 2023-03-22 17:56:39,407 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=40.25 KB/41220 for 876c388f3b400624afff0881af885f88 in 59ms, sequenceid=724, compaction requested=true 2023-03-22 17:56:39,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:39,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:39,407 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:39,407 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:39,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:39,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:39,408 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105470 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:39,408 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:39,408 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/A is initiating minor compaction (all files) 2023-03-22 17:56:39,408 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/B is initiating minor compaction (all files) 2023-03-22 17:56:39,408 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/A in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:39,408 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/B in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:39,408 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/591831f5e54c4771b6f0604471ea520c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/eb3acfc8bb154bb68f85efb77397245e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/1199c3adb24543de9b48b8c0c15516a3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9cbde7f301c946209466cbd47c4af4ab] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=46.5 K 2023-03-22 17:56:39,408 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4042ab1dd36f4a088de734f470c1ee34, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91ef6c7fca8f47aa9cc0af5d7379084b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b470c909202042b98607068221bd6ef5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/32e58fb00190484d9801efa2139a3e27] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=103.0 K 2023-03-22 17:56:39,408 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 591831f5e54c4771b6f0604471ea520c, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679507797479 2023-03-22 17:56:39,409 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 4042ab1dd36f4a088de734f470c1ee34, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679507797479 2023-03-22 17:56:39,409 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting eb3acfc8bb154bb68f85efb77397245e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679507797611 2023-03-22 17:56:39,409 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 91ef6c7fca8f47aa9cc0af5d7379084b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679507797611 2023-03-22 17:56:39,409 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 1199c3adb24543de9b48b8c0c15516a3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679507798677 2023-03-22 17:56:39,409 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting b470c909202042b98607068221bd6ef5, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679507798677 2023-03-22 17:56:39,409 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 9cbde7f301c946209466cbd47c4af4ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1679507799329 2023-03-22 17:56:39,409 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] compactions.Compactor(207): Compacting 32e58fb00190484d9801efa2139a3e27, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1679507799329 2023-03-22 17:56:39,425 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#B#compaction#658 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:39,425 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#A#compaction#657 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:39,432 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/cffde7cb343e42719597e0d42cefcff2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/cffde7cb343e42719597e0d42cefcff2 2023-03-22 17:56:39,435 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/B of 876c388f3b400624afff0881af885f88 into cffde7cb343e42719597e0d42cefcff2(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:39,435 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:39,435 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/B, priority=12, startTime=1679507799407; duration=0sec 2023-03-22 17:56:39,435 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-22 17:56:39,435 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-22 17:56:39,436 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-22 17:56:39,436 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1912): 876c388f3b400624afff0881af885f88/C is initiating minor compaction (all files) 2023-03-22 17:56:39,436 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 876c388f3b400624afff0881af885f88/C in TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:39,437 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/54981c27e5854c159c8b7872ff941fca, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/06b60623c7c6415fa0bd2103a438c78e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d2592b58f9e748b9ac8576b36fd0ae44, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/985d838f9c214df8a90ec8ca088e20c2] into tmpdir=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp, totalSize=46.5 K 2023-03-22 17:56:39,437 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 54981c27e5854c159c8b7872ff941fca, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1679507797479 2023-03-22 17:56:39,437 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 06b60623c7c6415fa0bd2103a438c78e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1679507797611 2023-03-22 17:56:39,437 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting d2592b58f9e748b9ac8576b36fd0ae44, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679507798677 2023-03-22 17:56:39,437 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] compactions.Compactor(207): Compacting 985d838f9c214df8a90ec8ca088e20c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=724, earliestPutTs=1679507799329 2023-03-22 17:56:39,442 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] throttle.PressureAwareThroughputController(145): 876c388f3b400624afff0881af885f88#C#compaction#659 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-22 17:56:39,496 DEBUG [Thread-4015] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-22 17:56:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-22 17:56:39,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34403] regionserver.HRegion(9158): Flush requested on 876c388f3b400624afff0881af885f88 2023-03-22 17:56:39,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 876c388f3b400624afff0881af885f88 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-22 17:56:39,759 DEBUG [Thread-4009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:55285 2023-03-22 17:56:39,759 DEBUG [Thread-4005] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:55285 2023-03-22 17:56:39,759 DEBUG [Thread-4009] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,759 DEBUG [Thread-4013] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:55285 2023-03-22 17:56:39,759 DEBUG [Thread-4005] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,759 DEBUG [Thread-4013] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=A 2023-03-22 17:56:39,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,760 DEBUG [Thread-4011] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:55285 2023-03-22 17:56:39,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=B 2023-03-22 17:56:39,760 DEBUG [Thread-4011] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 876c388f3b400624afff0881af885f88, store=C 2023-03-22 17:56:39,760 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 174 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 82 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(394): wrote 177 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 4489 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 4580 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 4881 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 4481 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(398): read 4600 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-22 17:56:39,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-22 17:56:39,761 INFO [Listener at localhost/44491] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-22 17:56:39,761 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:55285 2023-03-22 17:56:39,761 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:39,762 INFO [Listener at localhost/44491] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-22 17:56:39,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-22 17:56:39,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:39,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-22 17:56:39,766 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507799766"}]},"ts":"1679507799766"} 2023-03-22 17:56:39,767 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-22 17:56:39,769 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-22 17:56:39,769 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, UNASSIGN}] 2023-03-22 17:56:39,771 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, UNASSIGN 2023-03-22 17:56:39,771 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=876c388f3b400624afff0881af885f88, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:39,771 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507799771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679507799771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679507799771"}]},"ts":"1679507799771"} 2023-03-22 17:56:39,772 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723}] 2023-03-22 17:56:39,853 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/ec89726cf191432fa4a465122109a8a5 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/ec89726cf191432fa4a465122109a8a5 2023-03-22 17:56:39,856 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/C of 876c388f3b400624afff0881af885f88 into ec89726cf191432fa4a465122109a8a5(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-22 17:56:39,856 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:39,856 INFO [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/C, priority=12, startTime=1679507799407; duration=0sec 2023-03-22 17:56:39,856 DEBUG [RS:0;jenkins-hbase4:34403-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:39,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-22 17:56:39,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 876c388f3b400624afff0881af885f88 2023-03-22 17:56:39,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 876c388f3b400624afff0881af885f88, disabling compactions & flushes 2023-03-22 17:56:39,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:40,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-22 17:56:40,177 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230322dbecb3438392464a9c1a93f0c0f18205 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322dbecb3438392464a9c1a93f0c0f18205 2023-03-22 17:56:40,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-22 17:56:40,582 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=738, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/2073cbaf5118456191a18337d39ede72 2023-03-22 17:56:40,640 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/0554412813aa421186819cbf340e9d29 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/0554412813aa421186819cbf340e9d29 2023-03-22 17:56:40,643 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 876c388f3b400624afff0881af885f88/A of 876c388f3b400624afff0881af885f88 into 0554412813aa421186819cbf340e9d29(size=26.8 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-22 17:56:40,643 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:40,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:40,643 INFO [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88., storeName=876c388f3b400624afff0881af885f88/A, priority=12, startTime=1679507799407; duration=1sec 2023-03-22 17:56:40,643 DEBUG [RS:0;jenkins-hbase4:34403-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-22 17:56:40,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-22 17:56:40,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/9dddbac81dbb4b75b1e60f4999a42c40 2023-03-22 17:56:41,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/c8d02d2466c24db49155dbb2de52ff79 2023-03-22 17:56:41,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/A/2073cbaf5118456191a18337d39ede72 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/2073cbaf5118456191a18337d39ede72 2023-03-22 17:56:41,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/2073cbaf5118456191a18337d39ede72, entries=150, sequenceid=738, filesize=25.4 K 2023-03-22 17:56:41,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/B/9dddbac81dbb4b75b1e60f4999a42c40 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9dddbac81dbb4b75b1e60f4999a42c40 2023-03-22 17:56:41,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9dddbac81dbb4b75b1e60f4999a42c40, entries=150, sequenceid=738, filesize=11.9 K 2023-03-22 17:56:41,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/.tmp/C/c8d02d2466c24db49155dbb2de52ff79 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c8d02d2466c24db49155dbb2de52ff79 2023-03-22 17:56:41,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c8d02d2466c24db49155dbb2de52ff79, entries=150, sequenceid=738, filesize=11.9 K 2023-03-22 17:56:41,412 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 876c388f3b400624afff0881af885f88 in 1653ms, sequenceid=738, compaction requested=false 2023-03-22 17:56:41,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:41,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:41,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:41,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. after waiting 0 ms 2023-03-22 17:56:41,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:41,414 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/305f919884a74173b71561debf486d93, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1052d347c76c4f17b9dde0f7d8e0521d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/80453f10dd164ec08ae49c3d874c499a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/00f47cafd5274e95b3056b786fb6dd2b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f45a8cf5641e4074b846f5abbe4a1dfd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f66ce7060a5643a985c0b1a9966330a1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/400005421946496dad969d5d38089601, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c7bb560d59304908b798310e80b3c46d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/72f6b9174c4c40f5a78fc1f41f12e669, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e0bb0a167edc48019768c4f128dab058, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/26faa201b43f48a695f73f54f8f9ae1b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/907c9b33e5734c398d7b91d8912f8a6c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91489334b4854fa48a26dbd2cd771116, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/dc2aaf4052ab499dbf9a1ecb9f419e46, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b10c2cd8a11f4d6e988935f13d1f390d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b629286ff85c43609d48b41d011a1b60, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/9ef86991c36a467d9594a4410761b850, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6bce472614af408fa6d4073d3af09fae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6c8b8ceb2b1c48d084626758b55b8a72, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/92f9f9485c174e159bfe814f485e95b4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/7b66fbb8884140789b8d8ed97ae7ed8a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/15670c3aa4464d5e8bfae7cb5d8d6657, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/59643730766a4f378c16015780f85e22, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/17c3a3bf1eb64ca29cb1a62f9e3bcd0a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/3b0fa68b31dc4848a5b99f42745b2ce5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/aa70f8cb913f483ea3648a2f1ce4925f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4b0f2adbc4364909aee8f408cd648fa9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1f8c2c666beb45abb69e8c846dec92dd, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c1bf165b419943ecbca9be3b51bd7b7a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/a98035f7f867435cb886505fc2983ca1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1ac72d1072ce4057a807fd36c31c6306, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/55cbd74687694fed938a9d786cfa195e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e62d89936878417289903a6738768af1, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/687b3ba35f894473b36490ec06c2f75f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b1134614e15145f8980a8b0e86d2a235, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/0c238f866f3442a1be87429d345f0292, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/5e3f4c5c9174400fae88d484e01f79c8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/eff80674a15a4ab28ec95a2f218dbfda, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4042ab1dd36f4a088de734f470c1ee34, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91ef6c7fca8f47aa9cc0af5d7379084b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b470c909202042b98607068221bd6ef5, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/32e58fb00190484d9801efa2139a3e27] to archive 2023-03-22 17:56:41,415 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:56:41,416 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/305f919884a74173b71561debf486d93 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/305f919884a74173b71561debf486d93 2023-03-22 17:56:41,417 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1052d347c76c4f17b9dde0f7d8e0521d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1052d347c76c4f17b9dde0f7d8e0521d 2023-03-22 17:56:41,418 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/80453f10dd164ec08ae49c3d874c499a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/80453f10dd164ec08ae49c3d874c499a 2023-03-22 17:56:41,419 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/00f47cafd5274e95b3056b786fb6dd2b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/00f47cafd5274e95b3056b786fb6dd2b 2023-03-22 17:56:41,420 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f45a8cf5641e4074b846f5abbe4a1dfd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f45a8cf5641e4074b846f5abbe4a1dfd 2023-03-22 17:56:41,420 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f66ce7060a5643a985c0b1a9966330a1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/f66ce7060a5643a985c0b1a9966330a1 2023-03-22 17:56:41,421 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/400005421946496dad969d5d38089601 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/400005421946496dad969d5d38089601 2023-03-22 17:56:41,422 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c7bb560d59304908b798310e80b3c46d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c7bb560d59304908b798310e80b3c46d 2023-03-22 17:56:41,423 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/72f6b9174c4c40f5a78fc1f41f12e669 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/72f6b9174c4c40f5a78fc1f41f12e669 2023-03-22 17:56:41,423 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e0bb0a167edc48019768c4f128dab058 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e0bb0a167edc48019768c4f128dab058 2023-03-22 17:56:41,424 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/26faa201b43f48a695f73f54f8f9ae1b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/26faa201b43f48a695f73f54f8f9ae1b 2023-03-22 17:56:41,425 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/907c9b33e5734c398d7b91d8912f8a6c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/907c9b33e5734c398d7b91d8912f8a6c 2023-03-22 17:56:41,426 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91489334b4854fa48a26dbd2cd771116 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91489334b4854fa48a26dbd2cd771116 2023-03-22 17:56:41,426 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/dc2aaf4052ab499dbf9a1ecb9f419e46 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/dc2aaf4052ab499dbf9a1ecb9f419e46 2023-03-22 17:56:41,427 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b10c2cd8a11f4d6e988935f13d1f390d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b10c2cd8a11f4d6e988935f13d1f390d 2023-03-22 17:56:41,428 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b629286ff85c43609d48b41d011a1b60 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b629286ff85c43609d48b41d011a1b60 2023-03-22 17:56:41,429 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/9ef86991c36a467d9594a4410761b850 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/9ef86991c36a467d9594a4410761b850 2023-03-22 17:56:41,430 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6bce472614af408fa6d4073d3af09fae to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6bce472614af408fa6d4073d3af09fae 2023-03-22 17:56:41,431 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6c8b8ceb2b1c48d084626758b55b8a72 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/6c8b8ceb2b1c48d084626758b55b8a72 2023-03-22 17:56:41,431 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/92f9f9485c174e159bfe814f485e95b4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/92f9f9485c174e159bfe814f485e95b4 2023-03-22 17:56:41,432 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/7b66fbb8884140789b8d8ed97ae7ed8a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/7b66fbb8884140789b8d8ed97ae7ed8a 2023-03-22 17:56:41,433 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/15670c3aa4464d5e8bfae7cb5d8d6657 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/15670c3aa4464d5e8bfae7cb5d8d6657 2023-03-22 17:56:41,434 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/59643730766a4f378c16015780f85e22 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/59643730766a4f378c16015780f85e22 2023-03-22 17:56:41,434 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/17c3a3bf1eb64ca29cb1a62f9e3bcd0a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/17c3a3bf1eb64ca29cb1a62f9e3bcd0a 2023-03-22 17:56:41,435 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/3b0fa68b31dc4848a5b99f42745b2ce5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/3b0fa68b31dc4848a5b99f42745b2ce5 2023-03-22 17:56:41,436 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/aa70f8cb913f483ea3648a2f1ce4925f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/aa70f8cb913f483ea3648a2f1ce4925f 2023-03-22 17:56:41,437 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4b0f2adbc4364909aee8f408cd648fa9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4b0f2adbc4364909aee8f408cd648fa9 2023-03-22 17:56:41,438 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1f8c2c666beb45abb69e8c846dec92dd to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1f8c2c666beb45abb69e8c846dec92dd 2023-03-22 17:56:41,439 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c1bf165b419943ecbca9be3b51bd7b7a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/c1bf165b419943ecbca9be3b51bd7b7a 2023-03-22 17:56:41,439 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/a98035f7f867435cb886505fc2983ca1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/a98035f7f867435cb886505fc2983ca1 2023-03-22 17:56:41,440 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1ac72d1072ce4057a807fd36c31c6306 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/1ac72d1072ce4057a807fd36c31c6306 2023-03-22 17:56:41,441 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/55cbd74687694fed938a9d786cfa195e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/55cbd74687694fed938a9d786cfa195e 2023-03-22 17:56:41,442 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e62d89936878417289903a6738768af1 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/e62d89936878417289903a6738768af1 2023-03-22 17:56:41,443 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/687b3ba35f894473b36490ec06c2f75f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/687b3ba35f894473b36490ec06c2f75f 2023-03-22 17:56:41,458 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b1134614e15145f8980a8b0e86d2a235 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b1134614e15145f8980a8b0e86d2a235 2023-03-22 17:56:41,460 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/0c238f866f3442a1be87429d345f0292 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/0c238f866f3442a1be87429d345f0292 2023-03-22 17:56:41,461 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/5e3f4c5c9174400fae88d484e01f79c8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/5e3f4c5c9174400fae88d484e01f79c8 2023-03-22 17:56:41,462 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/eff80674a15a4ab28ec95a2f218dbfda to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/eff80674a15a4ab28ec95a2f218dbfda 2023-03-22 17:56:41,463 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4042ab1dd36f4a088de734f470c1ee34 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/4042ab1dd36f4a088de734f470c1ee34 2023-03-22 17:56:41,464 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91ef6c7fca8f47aa9cc0af5d7379084b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/91ef6c7fca8f47aa9cc0af5d7379084b 2023-03-22 17:56:41,464 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b470c909202042b98607068221bd6ef5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/b470c909202042b98607068221bd6ef5 2023-03-22 17:56:41,465 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/32e58fb00190484d9801efa2139a3e27 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/32e58fb00190484d9801efa2139a3e27 2023-03-22 17:56:41,469 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0115929ff37d4bbf938134ccff7256a9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f5a988776c0e4b3297dd8533d3a55606, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9a85eab2cc704fe7bc47baa83f27d7ae, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/5492ba0b905e49dcaf46cb95b805a88c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/80f513a154e249ea983f9b72444cfbf7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bba7a583f83b409faef865fcaab4b88a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/ec4b2cacc25748129477f4324bd33246, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/69c57dddaf8a46c08ed1bf6bf6931455, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/299bf2f4e3504270a8254e96963fa6f7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0649b3b5fe4f402bbcee328089c5aec7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b99618850fd44a02812ec587ac360ec8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d4d4632900a244afa2b0e364c0f3d9e0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4bec829fcbf9460db9c2176859ebafb0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/a02970dad27541019fd65defbc9f749d, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/dc4587e8b5a1476f93f5217b46252a25, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0395df31b14f46218236ecc27e56a4ce, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b2aebec835a444acbe91fd03b48e7d5a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b51dd99604c64a4a994b6d894eef1113, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d974f0c0da054a079a192c92ea488712, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/fad46194dca44bbbb7d3a6a6e0425807, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f2597f9906b1475b80ca034c18beb815, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4a64ce98c5ed4b01842e5e178321c196, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/db569ce875514055bebd655a411fb23b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3732390755a04a9f81a2f454da964cb4, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/e2f459d776664e51b07c6b60799381ba, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/96846c7d73e54017ab9edd923e8836c2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b85c4cf60982458f8c742411e651b7de, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2fe698ed6ed04e2c913adb0ad374e785, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/edeb0af0b16642a59d864fcb6b9d1eec, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d0191d66c60843b08b61b77cce5468ed, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/81893cc5ead84071b7c3e909965c7c0e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/af9d070eedfa4b1886e5dfefb4566e4f, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2528297abf074903a6f78860b69faff3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/7400c295c76e44b7840e9009c7a7ba48, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/86cb9406ef934e5a88d0777c0f118462, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/21c3647e8e3e4a168465072199a33e54, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3d26d023a6d5449e9108741f0eb910c0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/591831f5e54c4771b6f0604471ea520c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bd88762651d24c90ad2d1bca33349137, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/eb3acfc8bb154bb68f85efb77397245e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/1199c3adb24543de9b48b8c0c15516a3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9cbde7f301c946209466cbd47c4af4ab] to archive 2023-03-22 17:56:41,470 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:56:41,471 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0115929ff37d4bbf938134ccff7256a9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0115929ff37d4bbf938134ccff7256a9 2023-03-22 17:56:41,472 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f5a988776c0e4b3297dd8533d3a55606 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f5a988776c0e4b3297dd8533d3a55606 2023-03-22 17:56:41,473 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9a85eab2cc704fe7bc47baa83f27d7ae to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9a85eab2cc704fe7bc47baa83f27d7ae 2023-03-22 17:56:41,474 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/5492ba0b905e49dcaf46cb95b805a88c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/5492ba0b905e49dcaf46cb95b805a88c 2023-03-22 17:56:41,475 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/80f513a154e249ea983f9b72444cfbf7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/80f513a154e249ea983f9b72444cfbf7 2023-03-22 17:56:41,476 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bba7a583f83b409faef865fcaab4b88a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bba7a583f83b409faef865fcaab4b88a 2023-03-22 17:56:41,476 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/ec4b2cacc25748129477f4324bd33246 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/ec4b2cacc25748129477f4324bd33246 2023-03-22 17:56:41,477 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/69c57dddaf8a46c08ed1bf6bf6931455 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/69c57dddaf8a46c08ed1bf6bf6931455 2023-03-22 17:56:41,478 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/299bf2f4e3504270a8254e96963fa6f7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/299bf2f4e3504270a8254e96963fa6f7 2023-03-22 17:56:41,479 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0649b3b5fe4f402bbcee328089c5aec7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0649b3b5fe4f402bbcee328089c5aec7 2023-03-22 17:56:41,480 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b99618850fd44a02812ec587ac360ec8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b99618850fd44a02812ec587ac360ec8 2023-03-22 17:56:41,481 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d4d4632900a244afa2b0e364c0f3d9e0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d4d4632900a244afa2b0e364c0f3d9e0 2023-03-22 17:56:41,482 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4bec829fcbf9460db9c2176859ebafb0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4bec829fcbf9460db9c2176859ebafb0 2023-03-22 17:56:41,483 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/a02970dad27541019fd65defbc9f749d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/a02970dad27541019fd65defbc9f749d 2023-03-22 17:56:41,483 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/dc4587e8b5a1476f93f5217b46252a25 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/dc4587e8b5a1476f93f5217b46252a25 2023-03-22 17:56:41,484 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0395df31b14f46218236ecc27e56a4ce to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/0395df31b14f46218236ecc27e56a4ce 2023-03-22 17:56:41,485 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b2aebec835a444acbe91fd03b48e7d5a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b2aebec835a444acbe91fd03b48e7d5a 2023-03-22 17:56:41,486 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b51dd99604c64a4a994b6d894eef1113 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b51dd99604c64a4a994b6d894eef1113 2023-03-22 17:56:41,487 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d974f0c0da054a079a192c92ea488712 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d974f0c0da054a079a192c92ea488712 2023-03-22 17:56:41,488 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/fad46194dca44bbbb7d3a6a6e0425807 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/fad46194dca44bbbb7d3a6a6e0425807 2023-03-22 17:56:41,489 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f2597f9906b1475b80ca034c18beb815 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/f2597f9906b1475b80ca034c18beb815 2023-03-22 17:56:41,490 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4a64ce98c5ed4b01842e5e178321c196 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/4a64ce98c5ed4b01842e5e178321c196 2023-03-22 17:56:41,491 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/db569ce875514055bebd655a411fb23b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/db569ce875514055bebd655a411fb23b 2023-03-22 17:56:41,492 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3732390755a04a9f81a2f454da964cb4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3732390755a04a9f81a2f454da964cb4 2023-03-22 17:56:41,493 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/e2f459d776664e51b07c6b60799381ba to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/e2f459d776664e51b07c6b60799381ba 2023-03-22 17:56:41,493 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/96846c7d73e54017ab9edd923e8836c2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/96846c7d73e54017ab9edd923e8836c2 2023-03-22 17:56:41,494 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b85c4cf60982458f8c742411e651b7de to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/b85c4cf60982458f8c742411e651b7de 2023-03-22 17:56:41,495 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2fe698ed6ed04e2c913adb0ad374e785 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2fe698ed6ed04e2c913adb0ad374e785 2023-03-22 17:56:41,496 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/edeb0af0b16642a59d864fcb6b9d1eec to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/edeb0af0b16642a59d864fcb6b9d1eec 2023-03-22 17:56:41,497 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d0191d66c60843b08b61b77cce5468ed to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/d0191d66c60843b08b61b77cce5468ed 2023-03-22 17:56:41,498 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/81893cc5ead84071b7c3e909965c7c0e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/81893cc5ead84071b7c3e909965c7c0e 2023-03-22 17:56:41,499 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/af9d070eedfa4b1886e5dfefb4566e4f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/af9d070eedfa4b1886e5dfefb4566e4f 2023-03-22 17:56:41,500 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2528297abf074903a6f78860b69faff3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/2528297abf074903a6f78860b69faff3 2023-03-22 17:56:41,501 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/7400c295c76e44b7840e9009c7a7ba48 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/7400c295c76e44b7840e9009c7a7ba48 2023-03-22 17:56:41,502 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/86cb9406ef934e5a88d0777c0f118462 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/86cb9406ef934e5a88d0777c0f118462 2023-03-22 17:56:41,502 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/21c3647e8e3e4a168465072199a33e54 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/21c3647e8e3e4a168465072199a33e54 2023-03-22 17:56:41,503 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3d26d023a6d5449e9108741f0eb910c0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/3d26d023a6d5449e9108741f0eb910c0 2023-03-22 17:56:41,504 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/591831f5e54c4771b6f0604471ea520c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/591831f5e54c4771b6f0604471ea520c 2023-03-22 17:56:41,505 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bd88762651d24c90ad2d1bca33349137 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/bd88762651d24c90ad2d1bca33349137 2023-03-22 17:56:41,506 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/eb3acfc8bb154bb68f85efb77397245e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/eb3acfc8bb154bb68f85efb77397245e 2023-03-22 17:56:41,507 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/1199c3adb24543de9b48b8c0c15516a3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/1199c3adb24543de9b48b8c0c15516a3 2023-03-22 17:56:41,507 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9cbde7f301c946209466cbd47c4af4ab to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9cbde7f301c946209466cbd47c4af4ab 2023-03-22 17:56:41,511 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2db51ca0ea274d4383180b2323631769, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/db590b3d18d54dcba3b319790f42875b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/785cd1f4021749559c455e07ccdbab16, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7c5931d587cc4cfda78ba6e128321e70, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/68f4604c1daf4423bb11e6660444e705, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c94275372931452d862997cabc0ca6f0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/977f19a222f34be69142b2543a86bdf3, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2fbec198b0bd406ea9ae4104cd95cb8e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/73afeff9f0db44429968c30412653ec2, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/952b937dc8fd4f988df2662fd9ec232c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0926c76b8ae7466794f68bc4139005de, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d8c1d93e5d694272ba10799bded5429c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d5550e42ac924f348c542c34e9c881c7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f0a2fb49d6b34de0bd8f4824d17efc19, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/e58fae62c00f466bbb64a5420bde4ee0, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0763f78e1976466a81f9cb34d51aab85, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d9022796386a4f67abdcbbc2f0f12a06, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9191dee92662487086cc93cdbb44e0f6, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/716e2f5f42714415866fe36970629420, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9abb743892124ebfa062d4c88c7e00e7, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/355ed490c93e4f049de25c873e9c88cf, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/654d8c0177534374a6607bf2db7f4665, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/37f9715918654781a203c97a7206f0b8, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/38aa6899f02b4eb793ebbcc9417c0478, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6338c55cac46402fb33ed0a2536bdf25, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/11209c658e7742a5862687fb065ef98b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2dc288712eea48c08af07e1d1f94403c, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f5aa7f9889e84dc198c702de37eec345, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/252c56e0823445a2b0904efebebf714b, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6f41224f5ec94db184bf33c7481da4f9, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/cbf8c80a14354a58b9556d3e53a8e13a, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6cc7bdebd6f94d5f88970652fe1c5585, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f270309e123142a49ea04f935ce0f8fc, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7a4df21464f048108c579bf022c8b147, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d164a9b1a11244009bb508f8981a1feb, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/14341fefd9a2447eadb3d49983e31430, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2da092be0b584a54924651988e5f3837, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/54981c27e5854c159c8b7872ff941fca, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/1f249f90d76d4e69af20853ea1609136, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/06b60623c7c6415fa0bd2103a438c78e, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d2592b58f9e748b9ac8576b36fd0ae44, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/985d838f9c214df8a90ec8ca088e20c2] to archive 2023-03-22 17:56:41,511 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-22 17:56:41,512 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2db51ca0ea274d4383180b2323631769 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2db51ca0ea274d4383180b2323631769 2023-03-22 17:56:41,513 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/db590b3d18d54dcba3b319790f42875b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/db590b3d18d54dcba3b319790f42875b 2023-03-22 17:56:41,514 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/785cd1f4021749559c455e07ccdbab16 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/785cd1f4021749559c455e07ccdbab16 2023-03-22 17:56:41,515 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7c5931d587cc4cfda78ba6e128321e70 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7c5931d587cc4cfda78ba6e128321e70 2023-03-22 17:56:41,516 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/68f4604c1daf4423bb11e6660444e705 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/68f4604c1daf4423bb11e6660444e705 2023-03-22 17:56:41,517 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c94275372931452d862997cabc0ca6f0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c94275372931452d862997cabc0ca6f0 2023-03-22 17:56:41,517 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/977f19a222f34be69142b2543a86bdf3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/977f19a222f34be69142b2543a86bdf3 2023-03-22 17:56:41,518 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2fbec198b0bd406ea9ae4104cd95cb8e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2fbec198b0bd406ea9ae4104cd95cb8e 2023-03-22 17:56:41,519 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/73afeff9f0db44429968c30412653ec2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/73afeff9f0db44429968c30412653ec2 2023-03-22 17:56:41,519 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/952b937dc8fd4f988df2662fd9ec232c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/952b937dc8fd4f988df2662fd9ec232c 2023-03-22 17:56:41,520 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0926c76b8ae7466794f68bc4139005de to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0926c76b8ae7466794f68bc4139005de 2023-03-22 17:56:41,521 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d8c1d93e5d694272ba10799bded5429c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d8c1d93e5d694272ba10799bded5429c 2023-03-22 17:56:41,522 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d5550e42ac924f348c542c34e9c881c7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d5550e42ac924f348c542c34e9c881c7 2023-03-22 17:56:41,522 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f0a2fb49d6b34de0bd8f4824d17efc19 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f0a2fb49d6b34de0bd8f4824d17efc19 2023-03-22 17:56:41,523 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/e58fae62c00f466bbb64a5420bde4ee0 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/e58fae62c00f466bbb64a5420bde4ee0 2023-03-22 17:56:41,524 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0763f78e1976466a81f9cb34d51aab85 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/0763f78e1976466a81f9cb34d51aab85 2023-03-22 17:56:41,524 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d9022796386a4f67abdcbbc2f0f12a06 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d9022796386a4f67abdcbbc2f0f12a06 2023-03-22 17:56:41,525 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9191dee92662487086cc93cdbb44e0f6 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9191dee92662487086cc93cdbb44e0f6 2023-03-22 17:56:41,526 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/716e2f5f42714415866fe36970629420 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/716e2f5f42714415866fe36970629420 2023-03-22 17:56:41,526 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9abb743892124ebfa062d4c88c7e00e7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/9abb743892124ebfa062d4c88c7e00e7 2023-03-22 17:56:41,527 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/355ed490c93e4f049de25c873e9c88cf to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/355ed490c93e4f049de25c873e9c88cf 2023-03-22 17:56:41,528 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/654d8c0177534374a6607bf2db7f4665 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/654d8c0177534374a6607bf2db7f4665 2023-03-22 17:56:41,529 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/37f9715918654781a203c97a7206f0b8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/37f9715918654781a203c97a7206f0b8 2023-03-22 17:56:41,529 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/38aa6899f02b4eb793ebbcc9417c0478 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/38aa6899f02b4eb793ebbcc9417c0478 2023-03-22 17:56:41,530 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6338c55cac46402fb33ed0a2536bdf25 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6338c55cac46402fb33ed0a2536bdf25 2023-03-22 17:56:41,531 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/11209c658e7742a5862687fb065ef98b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/11209c658e7742a5862687fb065ef98b 2023-03-22 17:56:41,531 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2dc288712eea48c08af07e1d1f94403c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2dc288712eea48c08af07e1d1f94403c 2023-03-22 17:56:41,532 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f5aa7f9889e84dc198c702de37eec345 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f5aa7f9889e84dc198c702de37eec345 2023-03-22 17:56:41,533 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/252c56e0823445a2b0904efebebf714b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/252c56e0823445a2b0904efebebf714b 2023-03-22 17:56:41,534 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6f41224f5ec94db184bf33c7481da4f9 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6f41224f5ec94db184bf33c7481da4f9 2023-03-22 17:56:41,535 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/cbf8c80a14354a58b9556d3e53a8e13a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/cbf8c80a14354a58b9556d3e53a8e13a 2023-03-22 17:56:41,536 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6cc7bdebd6f94d5f88970652fe1c5585 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/6cc7bdebd6f94d5f88970652fe1c5585 2023-03-22 17:56:41,537 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f270309e123142a49ea04f935ce0f8fc to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/f270309e123142a49ea04f935ce0f8fc 2023-03-22 17:56:41,538 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7a4df21464f048108c579bf022c8b147 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/7a4df21464f048108c579bf022c8b147 2023-03-22 17:56:41,539 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d164a9b1a11244009bb508f8981a1feb to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d164a9b1a11244009bb508f8981a1feb 2023-03-22 17:56:41,540 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/14341fefd9a2447eadb3d49983e31430 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/14341fefd9a2447eadb3d49983e31430 2023-03-22 17:56:41,541 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2da092be0b584a54924651988e5f3837 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/2da092be0b584a54924651988e5f3837 2023-03-22 17:56:41,543 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/54981c27e5854c159c8b7872ff941fca to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/54981c27e5854c159c8b7872ff941fca 2023-03-22 17:56:41,544 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/1f249f90d76d4e69af20853ea1609136 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/1f249f90d76d4e69af20853ea1609136 2023-03-22 17:56:41,545 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/06b60623c7c6415fa0bd2103a438c78e to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/06b60623c7c6415fa0bd2103a438c78e 2023-03-22 17:56:41,545 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d2592b58f9e748b9ac8576b36fd0ae44 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/d2592b58f9e748b9ac8576b36fd0ae44 2023-03-22 17:56:41,546 DEBUG [StoreCloser-TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/985d838f9c214df8a90ec8ca088e20c2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/985d838f9c214df8a90ec8ca088e20c2 2023-03-22 17:56:41,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/recovered.edits/743.seqid, newMaxSeqId=743, maxSeqId=4 2023-03-22 17:56:41,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88. 2023-03-22 17:56:41,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 876c388f3b400624afff0881af885f88: 2023-03-22 17:56:41,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 876c388f3b400624afff0881af885f88 2023-03-22 17:56:41,553 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=876c388f3b400624afff0881af885f88, regionState=CLOSED 2023-03-22 17:56:41,553 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679507801553"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679507801553"}]},"ts":"1679507801553"} 2023-03-22 17:56:41,555 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-22 17:56:41,555 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 876c388f3b400624afff0881af885f88, server=jenkins-hbase4.apache.org,34403,1679507617723 in 1.7820 sec 2023-03-22 17:56:41,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-22 17:56:41,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=876c388f3b400624afff0881af885f88, UNASSIGN in 1.7860 sec 2023-03-22 17:56:41,556 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679507801556"}]},"ts":"1679507801556"} 2023-03-22 17:56:41,557 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-22 17:56:41,558 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-22 17:56:41,560 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7970 sec 2023-03-22 17:56:41,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-22 17:56:41,873 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-22 17:56:41,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-22 17:56:41,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:41,876 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:41,876 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:41,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-22 17:56:41,880 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 2023-03-22 17:56:41,881 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C, FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/recovered.edits] 2023-03-22 17:56:41,884 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/0554412813aa421186819cbf340e9d29 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/0554412813aa421186819cbf340e9d29 2023-03-22 17:56:41,885 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/2073cbaf5118456191a18337d39ede72 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/A/2073cbaf5118456191a18337d39ede72 2023-03-22 17:56:41,887 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9dddbac81dbb4b75b1e60f4999a42c40 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/9dddbac81dbb4b75b1e60f4999a42c40 2023-03-22 17:56:41,888 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/cffde7cb343e42719597e0d42cefcff2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/B/cffde7cb343e42719597e0d42cefcff2 2023-03-22 17:56:41,890 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c8d02d2466c24db49155dbb2de52ff79 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/c8d02d2466c24db49155dbb2de52ff79 2023-03-22 17:56:41,891 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/ec89726cf191432fa4a465122109a8a5 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/C/ec89726cf191432fa4a465122109a8a5 2023-03-22 17:56:41,893 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/recovered.edits/743.seqid to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88/recovered.edits/743.seqid 2023-03-22 17:56:41,894 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/.tmp/data/default/TestAcidGuarantees/876c388f3b400624afff0881af885f88 2023-03-22 17:56:41,894 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-22 17:56:41,894 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-22 17:56:41,895 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-22 17:56:41,897 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322039b894867b0482ab25eb4b31e221f5a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322039b894867b0482ab25eb4b31e221f5a 2023-03-22 17:56:41,898 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032208a6229d402148afafc6f61910d2c972 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032208a6229d402148afafc6f61910d2c972 2023-03-22 17:56:41,899 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032209085a59b4b74f48a14755905ccd4a8d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032209085a59b4b74f48a14755905ccd4a8d 2023-03-22 17:56:41,900 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322129bc996ae9941878d3ed267f3996225 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322129bc996ae9941878d3ed267f3996225 2023-03-22 17:56:41,901 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032214691f9e1a7645f983c2e112b49a6746 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032214691f9e1a7645f983c2e112b49a6746 2023-03-22 17:56:41,901 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032217a43faa6669439381d7d39675c22c66 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032217a43faa6669439381d7d39675c22c66 2023-03-22 17:56:41,902 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032218507229980141b4a9cd396cd96c52c3 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032218507229980141b4a9cd396cd96c52c3 2023-03-22 17:56:41,903 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032238b7e901ddc24718b380374285e72b47 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032238b7e901ddc24718b380374285e72b47 2023-03-22 17:56:41,904 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032245ce12b1caaf418087e6b5f915e82343 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032245ce12b1caaf418087e6b5f915e82343 2023-03-22 17:56:41,905 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303224f8dea7620154323b8704325a2033d22 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303224f8dea7620154323b8704325a2033d22 2023-03-22 17:56:41,906 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322578016ee11734746aa962cf72b5550b7 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322578016ee11734746aa962cf72b5550b7 2023-03-22 17:56:41,907 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303228244694645ef48b49aabcd57b89b6b1b to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303228244694645ef48b49aabcd57b89b6b1b 2023-03-22 17:56:41,908 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322828458341b5d44d1a108a079d05b199d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322828458341b5d44d1a108a079d05b199d 2023-03-22 17:56:41,909 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032285bdd27bb99d42c8a94c20110fbe282f to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032285bdd27bb99d42c8a94c20110fbe282f 2023-03-22 17:56:41,910 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032299f87b653d3b4c5d935008cc6baa3e8d to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032299f87b653d3b4c5d935008cc6baa3e8d 2023-03-22 17:56:41,911 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a28cb2509e5f4b1f99e2f4b9015cc2c8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322a28cb2509e5f4b1f99e2f4b9015cc2c8 2023-03-22 17:56:41,912 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b06442b52e3941d99fa6a9dfa3826e31 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b06442b52e3941d99fa6a9dfa3826e31 2023-03-22 17:56:41,913 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b2eda7d422c246f29620e43acd68d884 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b2eda7d422c246f29620e43acd68d884 2023-03-22 17:56:41,913 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b8c1ab014cf242a2b8342161bfaeeb12 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322b8c1ab014cf242a2b8342161bfaeeb12 2023-03-22 17:56:41,914 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c6f572f96c784ff3a63a8ed619c6d264 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322c6f572f96c784ff3a63a8ed619c6d264 2023-03-22 17:56:41,915 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ca86ffda1e994be18bfac3ea6dd67cd8 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ca86ffda1e994be18bfac3ea6dd67cd8 2023-03-22 17:56:41,916 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ce3683ec70c64c28bae1d4954e647181 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ce3683ec70c64c28bae1d4954e647181 2023-03-22 17:56:41,917 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322d2e9bf6567664954943071033fcc0352 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322d2e9bf6567664954943071033fcc0352 2023-03-22 17:56:41,918 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322dbecb3438392464a9c1a93f0c0f18205 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322dbecb3438392464a9c1a93f0c0f18205 2023-03-22 17:56:41,919 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e4bbbad0804549cc9773a4a645c90558 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e4bbbad0804549cc9773a4a645c90558 2023-03-22 17:56:41,919 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e9b5459d30ab47d5a79e949105a8ac17 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322e9b5459d30ab47d5a79e949105a8ac17 2023-03-22 17:56:41,920 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ed14911bfce042e8bacbebbbaec4df0c to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ed14911bfce042e8bacbebbbaec4df0c 2023-03-22 17:56:41,921 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ed4ab677b75b42fc94b7726d1999435a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322ed4ab677b75b42fc94b7726d1999435a 2023-03-22 17:56:41,922 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322efad86061fdf441194a21a795d436ef2 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322efad86061fdf441194a21a795d436ef2 2023-03-22 17:56:41,923 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f4c740363a8540bea51bc3b535d68023 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f4c740363a8540bea51bc3b535d68023 2023-03-22 17:56:41,924 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f72684aab28f4351bd8a8d95166182b4 to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322f72684aab28f4351bd8a8d95166182b4 2023-03-22 17:56:41,924 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322fb113555e2e341859d9ae532427af8db to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322fb113555e2e341859d9ae532427af8db 2023-03-22 17:56:41,925 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322feeb8e2531e64e08bcd1bf032231f23a to hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230322feeb8e2531e64e08bcd1bf032231f23a 2023-03-22 17:56:41,926 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-22 17:56:41,927 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:41,929 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-22 17:56:41,930 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-22 17:56:41,931 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:41,931 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-22 17:56:41,931 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679507801931"}]},"ts":"9223372036854775807"} 2023-03-22 17:56:41,932 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-22 17:56:41,932 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 876c388f3b400624afff0881af885f88, NAME => 'TestAcidGuarantees,,1679507776281.876c388f3b400624afff0881af885f88.', STARTKEY => '', ENDKEY => ''}] 2023-03-22 17:56:41,932 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-22 17:56:41,932 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679507801932"}]},"ts":"9223372036854775807"} 2023-03-22 17:56:41,933 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-22 17:56:41,936 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-22 17:56:41,936 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 62 msec 2023-03-22 17:56:41,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-22 17:56:41,978 INFO [Listener at localhost/44491] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-22 17:56:41,989 INFO [Listener at localhost/44491] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=238 (was 238), OpenFileDescriptor=637 (was 637), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=375 (was 350) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6309 (was 6174) - AvailableMemoryMB LEAK? - 2023-03-22 17:56:41,989 INFO [Listener at localhost/44491] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-22 17:56:41,989 INFO [Listener at localhost/44491] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-22 17:56:41,990 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:55285 2023-03-22 17:56:41,990 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:41,990 DEBUG [Listener at localhost/44491] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-22 17:56:41,990 DEBUG [Listener at localhost/44491] util.JVMClusterUtil(257): Found active master hash=380586161, stopped=false 2023-03-22 17:56:41,990 INFO [Listener at localhost/44491] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:56:41,992 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-22 17:56:41,992 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-22 17:56:41,992 INFO [Listener at localhost/44491] procedure2.ProcedureExecutor(629): Stopping 2023-03-22 17:56:41,992 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:56:41,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-22 17:56:41,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-22 17:56:41,992 DEBUG [Listener at localhost/44491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c57173e to 127.0.0.1:55285 2023-03-22 17:56:41,992 DEBUG [Listener at localhost/44491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:41,993 INFO [Listener at localhost/44491] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34403,1679507617723' ***** 2023-03-22 17:56:41,993 INFO [Listener at localhost/44491] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-22 17:56:41,993 INFO [RS:0;jenkins-hbase4:34403] regionserver.HeapMemoryManager(220): Stopping 2023-03-22 17:56:41,993 INFO [RS:0;jenkins-hbase4:34403] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-22 17:56:41,993 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-22 17:56:41,993 INFO [RS:0;jenkins-hbase4:34403] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-22 17:56:41,993 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(3304): Received CLOSE for e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:56:41,994 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:41,994 DEBUG [RS:0;jenkins-hbase4:34403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17c04ca6 to 127.0.0.1:55285 2023-03-22 17:56:41,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e8ac5092d76a6dd1e148cbdfe0f07096, disabling compactions & flushes 2023-03-22 17:56:41,995 DEBUG [RS:0;jenkins-hbase4:34403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:41,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:56:41,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:56:41,995 INFO [RS:0;jenkins-hbase4:34403] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-22 17:56:41,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. after waiting 0 ms 2023-03-22 17:56:41,995 INFO [RS:0;jenkins-hbase4:34403] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-22 17:56:41,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:56:41,995 INFO [RS:0;jenkins-hbase4:34403] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-22 17:56:41,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e8ac5092d76a6dd1e148cbdfe0f07096 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-22 17:56:41,995 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-22 17:56:41,995 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-22 17:56:41,996 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, e8ac5092d76a6dd1e148cbdfe0f07096=hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096.} 2023-03-22 17:56:41,996 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-22 17:56:41,996 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-22 17:56:41,996 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-22 17:56:41,996 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-22 17:56:41,996 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-22 17:56:41,996 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-22 17:56:41,997 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1505): Waiting on 1588230740, e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:56:41,998 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-22 17:56:42,002 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/.tmp/info/03ba0ee0fb81461985abc2fd43846a2f 2023-03-22 17:56:42,007 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 03ba0ee0fb81461985abc2fd43846a2f 2023-03-22 17:56:42,198 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1505): Waiting on 1588230740, e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:56:42,398 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1505): Waiting on 1588230740, e8ac5092d76a6dd1e148cbdfe0f07096 2023-03-22 17:56:42,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096/.tmp/info/f5975345bf244a83b2b6525b16b480e4 2023-03-22 17:56:42,406 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096/.tmp/info/f5975345bf244a83b2b6525b16b480e4 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096/info/f5975345bf244a83b2b6525b16b480e4 2023-03-22 17:56:42,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096/info/f5975345bf244a83b2b6525b16b480e4, entries=2, sequenceid=6, filesize=4.8 K 2023-03-22 17:56:42,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e8ac5092d76a6dd1e148cbdfe0f07096 in 415ms, sequenceid=6, compaction requested=false 2023-03-22 17:56:42,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/namespace/e8ac5092d76a6dd1e148cbdfe0f07096/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-22 17:56:42,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:56:42,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e8ac5092d76a6dd1e148cbdfe0f07096: 2023-03-22 17:56:42,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679507620403.e8ac5092d76a6dd1e148cbdfe0f07096. 2023-03-22 17:56:42,415 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/.tmp/rep_barrier/2b97c2915b184e64ac0171f28ca0943e 2023-03-22 17:56:42,418 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2b97c2915b184e64ac0171f28ca0943e 2023-03-22 17:56:42,454 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-22 17:56:42,454 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-22 17:56:42,598 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-22 17:56:42,799 DEBUG [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-22 17:56:42,823 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/.tmp/table/6889af3f95154775b664682a57c026b3 2023-03-22 17:56:42,826 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6889af3f95154775b664682a57c026b3 2023-03-22 17:56:42,827 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/.tmp/info/03ba0ee0fb81461985abc2fd43846a2f as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/info/03ba0ee0fb81461985abc2fd43846a2f 2023-03-22 17:56:42,830 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 03ba0ee0fb81461985abc2fd43846a2f 2023-03-22 17:56:42,830 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/info/03ba0ee0fb81461985abc2fd43846a2f, entries=22, sequenceid=93, filesize=7.1 K 2023-03-22 17:56:42,831 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/.tmp/rep_barrier/2b97c2915b184e64ac0171f28ca0943e as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/rep_barrier/2b97c2915b184e64ac0171f28ca0943e 2023-03-22 17:56:42,834 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2b97c2915b184e64ac0171f28ca0943e 2023-03-22 17:56:42,834 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/rep_barrier/2b97c2915b184e64ac0171f28ca0943e, entries=6, sequenceid=93, filesize=5.5 K 2023-03-22 17:56:42,835 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/.tmp/table/6889af3f95154775b664682a57c026b3 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/table/6889af3f95154775b664682a57c026b3 2023-03-22 17:56:42,837 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6889af3f95154775b664682a57c026b3 2023-03-22 17:56:42,838 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/table/6889af3f95154775b664682a57c026b3, entries=9, sequenceid=93, filesize=5.5 K 2023-03-22 17:56:42,838 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 842ms, sequenceid=93, compaction requested=false 2023-03-22 17:56:42,842 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-22 17:56:42,842 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-22 17:56:42,844 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-22 17:56:42,844 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-22 17:56:42,844 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-22 17:56:42,999 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34403,1679507617723; all regions closed. 2023-03-22 17:56:43,006 DEBUG [RS:0;jenkins-hbase4:34403] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/oldWALs 2023-03-22 17:56:43,006 INFO [RS:0;jenkins-hbase4:34403] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34403%2C1679507617723.meta:.meta(num 1679507620129) 2023-03-22 17:56:43,014 DEBUG [RS:0;jenkins-hbase4:34403] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/oldWALs 2023-03-22 17:56:43,014 INFO [RS:0;jenkins-hbase4:34403] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34403%2C1679507617723:(num 1679507619603) 2023-03-22 17:56:43,014 DEBUG [RS:0;jenkins-hbase4:34403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:43,014 INFO [RS:0;jenkins-hbase4:34403] regionserver.LeaseManager(133): Closed leases 2023-03-22 17:56:43,014 INFO [RS:0;jenkins-hbase4:34403] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-22 17:56:43,014 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-22 17:56:43,016 INFO [RS:0;jenkins-hbase4:34403] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34403 2023-03-22 17:56:43,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34403,1679507617723 2023-03-22 17:56:43,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-22 17:56:43,022 ERROR [Listener at localhost/44491-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20566bd4 rejected from java.util.concurrent.ThreadPoolExecutor@2469eb06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2754] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-22 17:56:43,022 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-22 17:56:43,022 ERROR [Listener at localhost/44491-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43dbcf84 rejected from java.util.concurrent.ThreadPoolExecutor@2469eb06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2754] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-22 17:56:43,023 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34403,1679507617723] 2023-03-22 17:56:43,024 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34403,1679507617723; numProcessing=1 2023-03-22 17:56:43,025 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34403,1679507617723 already deleted, retry=false 2023-03-22 17:56:43,026 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34403,1679507617723 expired; onlineServers=0 2023-03-22 17:56:43,026 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40203,1679507616414' ***** 2023-03-22 17:56:43,026 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-22 17:56:43,026 DEBUG [M:0;jenkins-hbase4:40203] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5792428a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-22 17:56:43,026 INFO [M:0;jenkins-hbase4:40203] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40203,1679507616414 2023-03-22 17:56:43,026 INFO [M:0;jenkins-hbase4:40203] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40203,1679507616414; all regions closed. 2023-03-22 17:56:43,026 DEBUG [M:0;jenkins-hbase4:40203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-22 17:56:43,026 DEBUG [M:0;jenkins-hbase4:40203] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-22 17:56:43,026 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-22 17:56:43,026 DEBUG [M:0;jenkins-hbase4:40203] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-22 17:56:43,026 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679507619263] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679507619263,5,FailOnTimeoutGroup] 2023-03-22 17:56:43,026 INFO [M:0;jenkins-hbase4:40203] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-22 17:56:43,026 INFO [M:0;jenkins-hbase4:40203] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-22 17:56:43,026 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679507619263] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679507619263,5,FailOnTimeoutGroup] 2023-03-22 17:56:43,027 INFO [M:0;jenkins-hbase4:40203] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-22 17:56:43,027 DEBUG [M:0;jenkins-hbase4:40203] master.HMaster(1512): Stopping service threads 2023-03-22 17:56:43,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-22 17:56:43,027 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-22 17:56:43,027 INFO [M:0;jenkins-hbase4:40203] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-22 17:56:43,028 ERROR [M:0;jenkins-hbase4:40203] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT 2023-03-22 17:56:43,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-22 17:56:43,028 INFO [M:0;jenkins-hbase4:40203] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-22 17:56:43,029 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-22 17:56:43,029 DEBUG [M:0;jenkins-hbase4:40203] zookeeper.ZKUtil(398): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-22 17:56:43,029 WARN [M:0;jenkins-hbase4:40203] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-22 17:56:43,029 INFO [M:0;jenkins-hbase4:40203] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-22 17:56:43,029 INFO [M:0;jenkins-hbase4:40203] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-22 17:56:43,030 DEBUG [M:0;jenkins-hbase4:40203] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-22 17:56:43,030 INFO [M:0;jenkins-hbase4:40203] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-22 17:56:43,030 DEBUG [M:0;jenkins-hbase4:40203] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-22 17:56:43,030 DEBUG [M:0;jenkins-hbase4:40203] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-22 17:56:43,030 DEBUG [M:0;jenkins-hbase4:40203] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-22 17:56:43,030 INFO [M:0;jenkins-hbase4:40203] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.84 KB heapSize=393.55 KB 2023-03-22 17:56:43,124 INFO [RS:0;jenkins-hbase4:34403] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34403,1679507617723; zookeeper connection closed. 2023-03-22 17:56:43,124 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-22 17:56:43,124 ERROR [Listener at localhost/44491-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b1af435 rejected from java.util.concurrent.ThreadPoolExecutor@2469eb06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2754] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-22 17:56:43,124 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7aa9b491] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7aa9b491 2023-03-22 17:56:43,124 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): regionserver:34403-0x1010d5c59010001, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-22 17:56:43,124 ERROR [Listener at localhost/44491-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c7ec6e3 rejected from java.util.concurrent.ThreadPoolExecutor@2469eb06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2754] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-22 17:56:43,125 INFO [Listener at localhost/44491] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-22 17:56:43,440 INFO [M:0;jenkins-hbase4:40203] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.84 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/0dc52b8f645b4ac28fa480f5b93963d2 2023-03-22 17:56:43,443 INFO [M:0;jenkins-hbase4:40203] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0dc52b8f645b4ac28fa480f5b93963d2 2023-03-22 17:56:43,444 DEBUG [M:0;jenkins-hbase4:40203] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/0dc52b8f645b4ac28fa480f5b93963d2 as hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0dc52b8f645b4ac28fa480f5b93963d2 2023-03-22 17:56:43,446 INFO [M:0;jenkins-hbase4:40203] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0dc52b8f645b4ac28fa480f5b93963d2 2023-03-22 17:56:43,446 INFO [M:0;jenkins-hbase4:40203] regionserver.HStore(1080): Added hdfs://localhost:40993/user/jenkins/test-data/2b744893-3248-f6f9-63ef-2fcf8964f9a7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0dc52b8f645b4ac28fa480f5b93963d2, entries=65, sequenceid=578, filesize=24.6 K 2023-03-22 17:56:43,447 INFO [M:0;jenkins-hbase4:40203] regionserver.HRegion(2948): Finished flush of dataSize ~344.84 KB/353113, heapSize ~393.53 KB/402976, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 417ms, sequenceid=578, compaction requested=false 2023-03-22 17:56:43,448 INFO [M:0;jenkins-hbase4:40203] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-22 17:56:43,448 DEBUG [M:0;jenkins-hbase4:40203] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-22 17:56:43,451 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-22 17:56:43,451 INFO [M:0;jenkins-hbase4:40203] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-22 17:56:43,452 INFO [M:0;jenkins-hbase4:40203] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40203 2023-03-22 17:56:43,453 DEBUG [M:0;jenkins-hbase4:40203] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40203,1679507616414 already deleted, retry=false 2023-03-22 17:56:43,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-22 17:56:43,556 INFO [M:0;jenkins-hbase4:40203] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40203,1679507616414; zookeeper connection closed. 2023-03-22 17:56:43,556 ERROR [Listener at localhost/44491-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26499f9f rejected from java.util.concurrent.ThreadPoolExecutor@47c420c4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4603] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-22 17:56:43,556 DEBUG [Listener at localhost/44491-EventThread] zookeeper.ZKWatcher(600): master:40203-0x1010d5c59010000, quorum=127.0.0.1:55285, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-22 17:56:43,556 ERROR [Listener at localhost/44491-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@503be789 rejected from java.util.concurrent.ThreadPoolExecutor@47c420c4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4603] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-22 17:56:43,558 WARN [Listener at localhost/44491] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-22 17:56:43,563 INFO [Listener at localhost/44491] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-22 17:56:43,674 WARN [BP-1269829930-172.31.14.131-1679507613043 heartbeating to localhost/127.0.0.1:40993] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-22 17:56:43,674 WARN [BP-1269829930-172.31.14.131-1679507613043 heartbeating to localhost/127.0.0.1:40993] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1269829930-172.31.14.131-1679507613043 (Datanode Uuid b573734f-a9bd-42cc-b584-b3b5f76a1564) service to localhost/127.0.0.1:40993 2023-03-22 17:56:43,677 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/cluster_5f8c7e52-8c9d-3fc1-676b-63f4f87dd676/dfs/data/data1/current/BP-1269829930-172.31.14.131-1679507613043] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-22 17:56:43,677 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/03a38193-fb55-e515-a19d-bec59c903d61/cluster_5f8c7e52-8c9d-3fc1-676b-63f4f87dd676/dfs/data/data2/current/BP-1269829930-172.31.14.131-1679507613043] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-22 17:56:43,712 INFO [Listener at localhost/44491] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-22 17:56:43,882 INFO [Listener at localhost/44491] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-22 17:56:44,055 INFO [Listener at localhost/44491] hbase.HBaseTestingUtility(1293): Minicluster is down